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ABSTRACT 


Fear  of  the  real  or  perceived  consequences  of  receiving  a  bid  protest  exists.  U.S.  Navy 
contracting  officers  have  some  concern  of  protests.  This  concern  can  be  linked  to  certain 
consequences  on  acquisition  strategies.  There  is  enough  qualitative  and  quantitative 
empirical  evidence  to  suggest  that  fear  of  protest  can  impact  what  would  otherwise  be 
prudent  business  decisions.  The  greatest  concerns  are  a  few  instances  of  inappropriate 
uses  of  lowest  price  technically  acceptable  and  the  reduced  technical  evaluation 
effectiveness  attributed  to  fear  of  protests.  If  fear  waters  down  the  source  selection 
hindering  its  ability  to  distinguish  between  the  true  value  of  offers,  then  contracting 
officers  must  ask  themselves  why  go  through  the  trouble  of  a  best-value  source  selection? 
Could  contracting  officers  simply  award  to  the  low  bidder?  To  what  extent  is  the  set  of 
stringent  source  selection  rules  driving  the  acquisition  team  to  this  result  by  default  (i.e., 
regardless  of  source  selection  method  actually  employed)?  Thus,  for  the  sake  of  stringent, 
fairness-based  rules,  contracted  outcomes  can  be  compromised.  Whether  the  tradeoff  is 
prudent  remains  to  be  determined.  Further  research  is  needed  to  ascertain  these  other 
culprits,  then  compare  the  relative  effects  of  fear  of  protest  among  other  factors. 
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I.  INTRODUCTION 

“Men  go  to  far  greater  lengths  to  avoid  what  they  fear  than  to  obtain  what  they 

desire.” 

— Dan  Brown,  The  Da  Vinci  Code 

A.  BACKGROUND 

The  U.S.  government  is  the  “world’s  largest  buyer  of  products  and  services,”  according 
to  the  Small  Business  Administration  (SB A)  (2012,  p.  2).  From  pens  and  pencils  to  the  joint 
strike  fighter  and  littoral  combat  ship  (LCS),  over  $500  billion  in  goods  and  services  has  been 
purchased  annually  since  fiscal  year  (FY)  2007.  However,  starting  in  FY2008,  year-over-year 
contract  spending  has  been  declining  because  of  fiscal  constraints.  Early  estimates  by  the 
Government  Accountability  Office  (GAO)  project  FY2013  spending  to  be  approximately  $460 
billion,  which  is  down  from  $513  billion  spent  in  FY2012. 

In  an  environment  of  fiscal  prudence,  acquisition  professionals  are  charged  with  the 
proper  stewardship  of  taxpayer  funds.  The  Federal  Acquisition  Regulation  (FAR)  charges  the 
members  of  the  acquisition  team  with  exercising  “personal  initiative  and  sound  business 
judgment  in  providing  the  best  value  product  or  service  to  meet  the  customer’s  needs”  (FAR 
1.102(d)).  The  FAR  also  states,  “An  essential  consideration  in  every  aspect  of  the  system  is 
maintaining  the  public’s  trust.  Not  only  must  the  system  have  integrity,  but  the  actions  of  each 
member  of  the  team  must  reflect  integrity,  fairness,  and  openness”  (FAR  1.102-2(c)  (1)).  One 
mechanism  that  is  installed  in  the  system  to  ensure  integrity  and  fairness  is  the  protest  system. 

Protests  ensure  “that  entities  doing  business  with  the  government  can  air  their  complaints 
about  governmental  contracting  processes  and  obtain  relief’  (Manuel  &  Schwartz,  2011,  p.  3). 
Without  a  forum  for  businesses  to  air  these  complaints,  they  could  become  unwilling  to  do 
business  with  the  government.  Protests  also  address  the  FAR  requirements  of  integrity,  fairness, 
and  openness  by  providing  for  the  “accountability  of  procurement  officials  and  government 
agencies  by  highlighting  and  correcting  mistakes  and  misconduct”  (Manuel  &  Schwartz,  2011,  p. 
3). 
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According  to  FAR  33.101,  a  protest  is  “a  written  objection  by  an  interested  party” 
concerning 

•  A  solicitation  or  other  request  by  an  agency  for  offers  for  a  contract  for  the 
procurement  of  property  or  services. 

•  The  cancellation  of  the  solicitation  or  other  request. 

•  An  award  or  proposed  award  of  the  contract. 

•  A  termination  or  cancellation  of  an  award  of  the  contract,  if  the  written  objection 
contains  an  allegation  that  the  termination  or  cancellation  is  based  in  whole  or  in 
part  on  improprieties  concerning  the  award  of  the  contract. 

Only  “interested  parties”  can  file  a  protest.  FAR  33.101  defines  an  interested  party  as  “an 
actual  or  prospective  offeror  whose  direct  economic  interest  would  be  affected  by  the  award  of  a 
contract  or  by  the  failure  to  award  a  contract.”  Interested  parties  can  file  the  protest  at  any  one  of 
the  following  venues:  the  agency.  Government  Accountability  Office  (GAO),  or  the  U.S.  Court 
of  Federal  Claims. 

Agency  protests  must  be  received  either  before  “bid  opening  or  the  closing  date  for 
receipt  of  proposals”  for  issues  with  the  solicitation  (FAR  33.103(e)).  Unsuccessful  offerors  have 
10  days  to  file  a  protest  after  the  impropriety  is  known,  or  they  have  five  days  after  an 
unsuccessful  offeror  debrief  is  conducted.  Alternative  dispute  resolution  should  be  attempted 
prior  to  the  protest  filing.  Agencies  should  make  their  best  effort  to  resolve  these  protests  within 
35  days  (FAR  33.101(g)). 

Protests  to  the  GAO  comprise  the  majority  of  all  protests  filed.  The  same  time  line  for 
submission  of  agency  protests  applies  to  protests  to  the  GAO.  The  protestor  must  provide  a  copy 
to  “the  official  and  location  designated  in  the  solicitation  or,  in  the  absence  of  such  a  designation, 
to  the  contracting  officer,  so  it  is  received  no  later  than  1  day  after  the  protest  is  filed  with  the 
GAO”  (FAR  33.104  (a)(1)).  The  contracting  officer  must  immediately  suspend  performance  on 
the  contract  in  dispute  if  already  awarded  or  terminate  the  contract.  If  the  contract  has  not  been 
awarded,  the  contracting  officer  should  notify  those  offerors  in  the  competitive  range  (FAR 
33.104(b)(3)). 
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An  overall  summary  of  protest  decisions  from  the  GAO  can  be  found  in  Table  1.  For 
FY2013,  Congress  added  the  requirement  for  the  GAO  to  include  “a  summary  of  the  most 
prevalent  grounds  for  sustaining  protests”  (GAO,  2014).  During  2013,  the  most  prevalent 
reasons  for  sustainment  were 

•  failure  to  follow  the  solicitation  evaluation  criteria, 

•  inadequate  documentation  of  the  record, 

•  unequal  treatment  of  offerors,  and 

•  unreasonable  price  or  cost  evaluation  (GAO,  2014). 


Table  1.  Bid  Protest  Statistics  for  Fiscal  Years  2009-2013  (from  GAO, 

2014,  p.  4) 


FY2013 

FY2012 

FY2011 

FY2010 

FY2009 

Cases  Filed’ 

2429 

(do»in24ih 

2.475 
(up  5%) 

2.353 
(up  2%) 

2.299 
(up  16%) 

1.989 
(up  20%) 

Cases  Closed 

2.538® 

2.495 

2.292 

2.226 

1.920 

Merit  (Sustain  +  Deny) 
Decisions 

509 

570 

417 

441 

315 

Number  of  Sustains 

87 

106 

67 

82 

57 

Sustain  Rate 

17% 

18.6% 

16% 

19% 

18% 

Effecbveness  Rate* 

43% 

42% 

42% 

42% 

45% 

ADR®  (cases  used) 

145 

106 

140 

159 

149 

ADR  Success  Rate® 

8«% 

80% 

82% 

80% 

93% 

Hearings^ 

3.36% 

(31  cases) 

6.17% 

(56  cases) 

8% 

(46  cases) 

10% 

(61  cases) 

12% 

(65  cases) 

This  was  the  first  time  since  FY2009  that  protests  did  not  increase  year  over  year,  but 
these  numbers  can  be  misleading.  When  measured  against  the  number  of  contract  actions 
awarded,  the  number  of  protests  represents  less  than  one  percent.  However,  the  percentage  went 
up  slightly  for  FY2013.  In  FY2011  and  FY2012,  protests  represented  .014  percent  of  contracting 
actions  awarded  across  the  federal  government.  In  FY20I3,  that  number  rose  to  .018  percent  of 
federal  actions.  So,  while  the  overall  number  of  protests  filed  declined  by  two  percent,  the 
number  of  contract  actions  also  declined  by  23  percent.  According  to  these  numbers,  the  protest 
situation  got  slightly  worse. 
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No  agency  can  prevent  a  protest,  but  a  well-prepared  plan  can  minimize  potential  grounds 
for  protest  (Rumbaugh,  2010,  p.  427).  As  the  GAO  has  identified,  there  are  numerous  reasons 
why  an  interested  party  might  choose  to  file  a  protest: 

A  number  of  analysts  have  suggested  that  companies  are  increasingly  likely  to  file 
protests  when  it  is  in  their  business  interest  to  do  so,  even  when  they  do  not 
believe  there  was  an  error  in  the  procurement  process.  When  agencies  do  not 
adequately  debrief  losing  bidders,  the  losing  companies  may  file  a  protest  to 
determine  why  they  lost  the  competition.  Other  reasons  companies  may  protest, 
include  hoping  to  influence  the  outcome  of  future  competitions  (akin  to  “yelling 
at  the  referee”);  proving  to  shareholders  and  executive  managers  that  they  are 
doing  everything  they  can  to  win  contracts;  or  even  seeking  to  hurt  the 
competition  by  delaying  a  contract  award.  (Schwartz  et  ah,  2013,  p.  12) 

Even  though  the  GAO’s  sustainment  rate  is  minute  when  compared  to  the  number  of 
procurements  awarded  throughout  the  federal  government,  the  Department  of  Defense  (DOD) 
acquisition  workforce  thinks  that  it  is  crucial  to  avoid  protests  (Gordon,  2011).  An  agency’s  best 
policy  to  prevent  a  protest  is  to  mitigate  the  causes  of  the  protest.  This  desire  to  avoid  a  protest  is 
the  driving  force  behind  acquisition  decisions,  internal  and  external  policies,  and  resources 
needed  to  deliberately  or  subconsciously  remove  the  threat  of  a  protest.  Throughout  the 
remainder  of  this  study,  the  desire  or  priority  to  avoid  bid  protests  is  referred  to  as  the  fear  of 
protests. 

B.  PROBLEM  STATEMENT  AND  GAP  IN  LITERATURE 

According  to  three  Congressional  Research  Service  (CRS)  reports  (Manuel  &  Schwartz, 
2011;  Schwartz,  Manuel,  &  Martinez,  2013;  O’Rourke,  2014),  Gordon’s  publication  (2013),  and 
other  publications  (Knauth,  2013;  Maser  et  ah,  2010;  Kendall,  2012),  the  fear  of  protests  exist. 
Fear  exists  because  no  agency  can  prevent  a  protest,  and  protests  may  result  in  the  following: 

•  issuing  a  stop  work  order  to  suspend  performance, 

•  reevaluating  proposals, 

•  awarding  proposal  preparation  and  protest  filing  cost  to  the  successful  protester, 
or 

•  terminating  the  awarded  contract  and  re-soliciting  the  requirement  (Rumbaugh, 
2010,  p.  415). 
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These  protest  results  add  costs,  deplete  resources,  and  create  other  intangible  effects  (e.g., 
a  diminished  reputation)  to  an  agency.  Based  on  the  protest  results,  protests  could  be  considered 
a  problem  to  an  agency  or  procurement,  even  though  protests  are  an  avenue  in  ensuring  fairness, 
transparency,  and  accountability  in  the  federal  acquisition  system.  To  mitigate  the  concern  of 
protest,  agencies  change  their  acquisition  strategies  and  outcomes.  There  is  no  data  that  exist  to 
show  the  true  monetary  and  nonmonetary  effects  of  such  mitigation  as  a  result  of  fear. 

Frank  Kendall,  Undersecretary  of  Defense  for  Acquisition,  Technology  and  Logistics 
(AT&L),  spoke  at  a  Center  for  Strategic  and  International  Studies  (CSIS)  conference  in 
Washington,  DC.  The  moderator  asked  Mr.  Kendall, 

My  question — my  concern  goes  to  things  that  I  heard  when  I  was  in  the  White 
House,  but  also  what  I  heard  as  part  of  my  participation  on  the  procurement  round 
table,  and  that’s  the  fear  in  the  acquisition  work  force  on  the  part  of  our 
contracting  officers,  contract  specialists  and  what  that  fear  is  driving  them  to. 
(Kendall,  2012) 

In  response,  Kendall  acknowledged  that  fear  exists  within  the  DOD: 

The  fear  of  protest  does  exist  out  there.  It  kind  of  permeates  through  the  system. 

You  know,  I  think  at  the  end  of  the  day  we  should  not  be  paranoid — or  paralyzed, 
is  probably  a  better  word — by  fear  of  protest  or  by  fear  of  litigation.  (Kendall, 

2012) 

Kendall,  who  is  the  head  contracting  official  for  the  DOD,  alluded  to  paralysis  by  the  fear  of 
protest.  Evidence  is  presented  in  this  thesis  that  shows  the  causes  and  effects  of  this  paralysis. 

There  is  evidence  that  Navy  acquisition  professionals  and  senior  leaders  were  fearful  of  a 
protest  for  the  Navy’s  LCS.  According  to  the  Congressional  Research  Service  report,  it  was 
insinuated  that  the  desire  to  avoid  a  protest  situation  affected  key  decisions: 

What  role,  if  any,  did  a  desire  by  the  Navy  to  avoid  a  potential  contract  protest 
against  the  Navy’s  down  select  decision  play  in  the  Navy’s  decision  to  propose 
the  alternate  dual-award  strategy?  For  example,  how  concerned,  if  at  all,  was  the 
Navy  that  the  announcement  of  an  LCS  down  select  decision  might  lead  to  a 
contract  protest  and  controversy  somewhat  like  what  has  been  experienced  in  the 
Air  Force’s  KC-X  aerial  refueling  tanker  acquisition  program?  (O’Rourke,  2014, 

p.  68) 
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A  December  13,  2010,  press  report  on  the  LCS  program  stated,  “One  high-level  Navy 
source  recently  said  that  without  the  dual-ship  approach,  ‘there  is  100  percent  chance  of  a 
protest’”  (Cavas,  2010).  As  a  result  of  this  fear,  the  Navy  changed  its  acquisition  plan  from  a 
down  select  to  a  dual-award  strategy. 

While  scholars  and  the  GAO  have  identified  these  deleterious  effects  of  bid  protests  on 
the  government  (Gordon,  2013),  no  research  or  studies  to  date  have  quantified  them.  In  other 
words,  we  do  not  know  the  magnitude  of  fear  of  protests.  More  specifically,  we  do  not  know  the 
following:  the  extent  that  fear  of  a  protest  affects  acquisition  strategies,  the  monetary  and  non¬ 
monetary  costs  of  a  bid  protest,  and  the  lengths  that  DOD  acquisition  professionals  would  go  to 
avoid  a  protest.  According  to  Gordon,  in  turn,  the  overwhelming  culture  among  DOD  contracting 
officers  is  to  avoid  protests  to  the  maximum  extent  possible.  Contributing  to  the  level  of  fear  is 
the  fact  that  protests  often  impose  litigation  costs,  termination  costs,  transaction  costs,  and 
opportunity  costs  on  the  government;  add  to  the  workload  of  a  shrinking  acquisition  workforce; 
delay  the  time  of  contractor  performance  and  delivery;  and  bring  shame  to  the  source  selection 
team  (SST).  Even  when  a  bid  protest  is  denied,  it  usually  holds  up  the  protested  acquisition 
(Gordon,  2013). 

Gordon  went  on  to  talk  about  the  effects  of  protest  fear:  “Another  concern  about  the  cost 
of  the  protest  system  relates  to  what  might  be  called  its  indirect  impact.  Fear  of  protests  is  often 
given  as  the  explanation  for  contracting  officers’  preference  for  certain  courses  of  action  over 
others”  (Gordon,  2013,  p.  36).  According  to  one  GAO  report,  protest  fear  can  negatively 
influence  agency  behavior. 

Fear  of  protest  may  motivate  agency  officials  to  conduct  more  rigorous  market 
research,  hold  a  competition  instead  of  awarding  a  sole-source  contract,  or 
conduct  more  thorough  and  fair  competition.  On  the  other  hand,  fear  of  a  protest 
could  also  prompt  officials  to  try  to  structure  a  contract  in  a  manner  they  deem 
less  likely  to  be  protested,  such  as  using  lowest  price  technically  acceptable  award 
criteria  instead  of  a  best-value  competition.  (Schwartz  et  ah,  2013,  p.l2) 

When  acquisition  professionals  structure  a  solicitation  using  lowest  price  technically 
acceptable  (FPTA)  source  selection  methodology  instead  of  using  trade-offs,  they  could  be 
hurting  the  industry  and  the  customer.  “The  president  of  a  major  satellite  services  provider  said 
the  U.S.  military’s  ‘lowest  price,  technically  acceptable’  procurement  strategy  is  stifling 
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innovation  and  ultimately  shortchanging  war  fighters”  (Magnuson,  2014,  p.  1).  This  comment 
suggests  that  one  unintended  consequence  of  bid  protest  is  an  inappropriate  acquisition  strategy, 
namely  the  source  selection  method.  However,  no  quantifiable  data  exists  that  explores  the 
relationship  between  the  fear  of  protest  and  the  appropriateness  of  the  acquisition  strategy. 

This  research  will  address  the  causes  and  deleterious  effects  of  protest  fear,  and  examine 
the  effects  of  fear  on  the  acquisition  system.  The  research  will  analyze  the  data  to  determine 
whether  there  are  any  significant  correlations  between  the  cause  and  effects  of  fear,  and  we 
attempt  to  quantify  the  magnitude  of  fear  within  the  U.S.  Navy.  It  will  then  analyze  the  data  to 
present  ways  in  which  acquisition  leaders  across  the  DOD  can  mitigate  this  fear. 

C.  RESEARCH  OBJECTIVES 

The  objective  of  this  research  is  to  address  in  depth  the  true  causes  and  effects  of  bid 
protest  fear.  Specifically,  we  explore  the  following  research  questions: 

•  Are  contracting  officers  sufficiently  concerned  about  bid  protests  to  alter 
acquisition  strategies? 

•  If  so,  what  factors  affect  protest  fear? 

•  What  are  the  consequences  to  the  acquisition  strategy  and  contract  outcomes? 

D.  METHODOLOGY 

Because  of  the  nature  of  the  research  questions,  it  is  appropriate  to  use  a  combination  of 
qualitative  and  quantitative  methodologies.  Literature  from  several  governmental,  public,  and 
private  studies  and  reports  are  used  to  develop  the  conceptual  model,  hypotheses,  interview 
questionnaire,  and  survey.  The  existing  literature  is  used  to  identify  problems  that  have  not  been 
addressed — namely  the  magnitude,  antecedents  to,  and  consequences  of  a  fear  of  protest.  The 
existing  literature  left  an  opportunity  for  further  research  into  an  area  that  would  be  considered 
beneficial  to  the  acquisition  professionals  across  the  federal  government. 

To  ensure  that  this  research  explored  the  most  likely  and  predominant  cause  and  effect 
variables  of  a  fear  of  protest,  several  interviews  were  conducted  at  two  major  Navy  contracting 
activities.  The  goals  of  the  interviews  were  to  validate  the  conceptual  model,  identify  potentially 
key  omitted  variables,  and  validate  the  measures  of  variables  for  which  there  existed  no 
measurement  scales. 
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A  survey  was  deployed  online  to  a  population  of  U.S.  Navy  contracting  officers  and 
specialists.  The  data  was  then  used  to  perform  multiple  regression  analysis  of  causal 
relationships  per  the  conceptual  model. 

E.  ORGANIZATION  OF  THE  STUDY 

This  study  is  broken  up  into  several  chapters.  Chapter  II  is  the  literature  review.  This 
chapter  will  discuss  and  provide  an  overview  of  several  applicable  theories.  In  Chapter  III 
discusses  the  methodology  used,  which  includes  the  research  and  survey  designs,  interviews,  and 
construct  measurements.  The  data  results,  exploratory  factor  analysis,  and  data  regression  models 
are  discussed  in  Chapter  IV.  Finally,  the  findings  will  present  answers  to  the  research  questions, 
and  offer  managerial  implications  and  recommendations  in  Chapter  V. 

F.  SCOPE 

Due  to  time,  resources,  and  administrative  constraints,  this  research  was  limited  to  U.S. 
Navy  personnel.  Furthermore,  this  research  focused  on  both  military  and  civilian  contracting 
officers  who  make  up  the  Navy’s  acquisition  workforce  within  the  United  States  and  overseas 
locations.  This  research  represents  a  wide  variety  of  goods  and  services  procured  using  FAR  Part 
15  source  selection  methods  above  $150,000,  a  wide  variety  of  supported  units’  mission,  and  a 
range  of  contracting  commands  within  the  U.S.  Navy.  We  only  analyzed  formal  contracting 
practices  in  accordance  with  FAR  Part  15;  contract  action  awards  pursuant  to  FAR  Parts  13  and 
8  were  not  considered  since  the  probability  of  receiving  a  protest  of  simplified  and  delivery/task 
orders  is  lower.  Under  FAR  Part  8,  protests  are  lower  because  the  government  is  using 
mandatory  sources  for  products  and  services.  Under  FAR  Part  13, 

Unless  the  contract  action  requires  synopsis  pursuant  to  5.101  and  an  exception 
under  5.202  is  not  applicable,  consider  solicitation  of  at  least  three  sources  to 
promote  competition  to  the  maximum  extent  practicable.  Whenever  practicable, 
request  quotations  or  offers  from  two  sources  not  included  in  the  previous 
solicitation.  (FAR  13.104) 

Under  FAR  Part  15,  there  is  a  greater  chance  for  protest  because  of  the  effects  of  best 
value  (LPTA  and  trade-off)  under  full  and  open  competition. 
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G.  MANAGERIAL  AND  THEORETICAL  IMPLICATIONS 

A  study  that  explores  the  causes  and  effects  of  the  fear  of  protest  is  beneficial  to  DOD 
and  federal  government  acquisition  practitioners.  First,  if  the  fear  of  a  bid  protest  is  significant, 
source  selections  may  result  in  the  selection  of  a  sub-optimal  contractor,  thereby  compromising 
performance  and  buyer  satisfaction.  Alternatively,  agencies  may  award  excessive  contracts  to 
avoid  protests.  This  increases  transaction  costs  and  contradicts  strategic  sourcing  goals.  It  can 
also  unnecessarily  increase  contractors’  bid  and  proposal  costs  as  they  compete  for  individual 
task  orders  under  multiple-award  contracts  (e.g.,  indefinite  delivery,  indefinite  quantity  [IDIQ]). 
Additionally,  such  fear  may  also  lead  SSTs  to  substantially  increase  transaction  costs  as  extra 
measures  are  taken  to  avoid  bid  protests.  These  extra  measures  also  delay  receipt  of  contractor 
performance,  thereby  impeding  the  mission. 

Second,  the  acquisition  community  can  use  the  results  of  this  study  to  become  aware  of 
the  effects  of  such  fear  and  its  implications.  With  this  awareness,  the  acquisition  community  can 
decide  where  in  the  acquisition  process  to  allocate  additional  resources  to  reduce  fear  of  protest 
and  the  subsequent  consequences.  This  strategy  allows  the  acquisition  community  to  be 
proactive  instead  of  reactive  in  dealing  with  fear  and  its  consequences. 

Third,  acquisition  leaders  and  planners  within  the  community  can  use  the  information  to 
quantify  and  assess  the  amount  of  man  hours,  money,  and  people  (i.e.,  resources)  that  are  being 
consumed  because  of  protest  fear.  With  this  information,  acquisition  leaders  and  planners  can 
better  align  their  internal/external  policies,  acquisition  strategies,  and  resources  to  mitigate  the 
causes  and  effects  driving  the  fear  of  protest,  which  ultimately  can  save  money.  In  addition, 
acquisition  professionals  can  understand  and  quantify  the  resources  spent  on  source  selections. 
With  the  understanding  of  the  amount  of  human  capital,  time,  and  resources,  the  agency  can 
determine  which  internal/external  policies  and  acquisition  strategies  are  not  cost  effective. 
Lastly,  the  amount  of  human  capital,  time,  and  resources  spent  on  bid  protest  mitigation  is  an 
opportunity  cost  of  human  capital,  time,  and  resources  that  could  have  been  devoted  to  other 
critical  areas.  This  is  critical  for  leaders,  because  as  the  DOD  budget  gets  smaller,  human  capital, 
time,  and  resources  will  become  very  precious  commodities  that  agencies  cannot  afford  to  waste 
or  allocate  to  inopportune  activities. 
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Finally,  this  study  recommends  additional  research  into  the  fear  of  protest  across  different 
government  agencies.  Each  agency  has  different  acquisition  workforce  compositions,  cultures, 
acquisition  policies  and  strategies;  the  U.S.  Navy’s  perspective  can  only  be  used  as  a  point  of 
reference.  This  does  not  reflect  the  true  level  of  fear,  causes,  and  effects  across  the  federal 
government.  But  this  study  can  be  used  as  a  starting  point  for  further  research. 

H.  CONCLUSION 

In  conclusion,  this  study  examines  the  fear  of  a  bid  protest — a  phenomenon  purportedly 
having  a  significant  effect  on  acquisition  decisions.  This  chapter  discusses  the  problem 
statements  and  literature  gaps  along  with  the  methodology  that  describes  the  approach  to 
collecting  data  to  answer  the  research  questions.  The  following  chapter  discusses  applicable 
theories  along  with  the  research  hypotheses  and  the  conceptual  model. 
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II.  LITERATURE  REVIEW 


A.  INTRODUCTION 

The  Public  Value  Framework  (PVF;  Moore,  2000)  and  stakeholder  theories  can  be 
applied  to  the  fear  of  bid  protests  from  an  economic  and  managerial  perspective.  This  chapter 
provides  further  explanation  of  the  study’s  conceptual  model,  research  objectives,  variables,  and 
hypotheses.  A  visual  representation  of  the  model  and  a  summary  of  the  hypotheses  are  provided 
at  the  end  of  the  chapter. 

B.  BID  PROTESTS 

There  are  numerous  reasons  why  an  interested  party  might  file  a  bid  protest.  According  to 
a  2013  CRS  report. 

According  to  analysts,  the  most  common  government  errors  cited  in  protests  are 
poorly  written  or  vague  contract  requirements,  failure  to  follow  the  process  or 
criteria  laid  out  in  the  request  for  proposals,  and  failure  to  adequately  document 
their  findings.  Some  analysts  have  attributed  these  errors  to  an  inexperienced  or 
insufficiently  trained  acquisition  workforce.  (Schwartz  et  ah,  2013,  p.  11) 

Although  there  are  many  causes  for  protest,  the  most  typical  include 

•  improper  agency  evaluations, 

•  a  lack  of  meaningful  discussions, 

•  defective  solicitations, 

•  improper  exclusion  from  the  competitive  range, 

•  a  lack  of  cost  realism,  and 

•  agency  bias  or  bad  faith  (Rumbaugh,  2010). 

A  particular  business  could  file  a  protest  whether  the  protest  is  frivolous  or  not. 

The  threat  of  a  protest  can  influence  agency  acquisition  behaviors.  Instead  of  having  full 
and  open  competition,  end  user  customers  and  acquisition  professionals  could  structure  a  sole 
source  contract.  Instead  of  awarding  a  single  contract  (as  originally  intended  per  the  acquisition 
strategy),  multiple  award  contracts  are  sometimes  awarded  in  order  to  avoid  a  protest.  In 
instances  where  a  trade-off  source  selection  methodology  is  more  appropriate  in  order  to  attain 
the  best  value  to  the  government,  an  LPTA  methodology  is  sometimes  used.  Instead  of  limiting 
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proposals  in  the  competitive  range,  non-qualifying  or  non-competitive  proposals  are  retained. 
Instead  of  conducting  discussions,  agencies  may  award  without  discussions. 

Protests  come  with  monetary  and  non-monetary  costs.  An  agency  must  incur  costs  to 
prevent  a  potential  bid  protest  (e.g.,  thoroughly  documenting  and  substantiating  proposal 
evaluations  and  trade-off  decisions),  to  defend  against  an  actual  protest,  and  to  take  corrective 
actions.  The  end  users  bear  costs  as  well,  since  their  requirements  are  delayed  or  jeopardized.  For 
example,  building,  fielding,  and  sustaining  two  varieties  of  LCS  platforms  are  substantially  more 
costly  than  doing  so  for  one  variety.  All  parties  involved  in  a  bid  protest  incur  the  opportunity 
costs  of  their  best  choice.  For  example,  an  agency  allocating  time  and  people  to  prevent  a  protest 
diverts  time  and  people  from  other  critical  requirements.  The  time  and  peoples’  salaries  or  wages 
are  the  opportunity  cost  for  the  critical  requirements. 

C.  PUBLIC  VALUE  FRAMEWORK 

PVF  was  introduced  by  Harvard  professor  Mark  H.  Moore  and  has  been  used  to  evaluate 
and  identify  value  in,  mainly,  the  public  sector.  Value  in  the  public  sector  is  much  different  than 
it  is  in  the  private  sector.  Often  in  the  private  sector,  industry  uses  shareholder  value  as  a  means 
of  evaluating  itself.  The  private  sector,  however,  is  much  different.  PVF  has  been  utilized  to  “get 
public  managers  thinking  about  what  is  most  valuable  in  the  service  that  they  run  and  to  consider 
how  effective  management  can  make  the  service  the  best  that  it  can  be”  (Coats  &  Passmore, 
2008,  p.  4). 

PVF  for  application  in  the  government  sector  can  be  explained  by  the  strategic  triangle 
(Heymann,  1987;  Moore,  1995).  These  three  elements  are  public  value,  legitimacy  support,  and 
operational  capability.  In  contrast  to  private  sector  operations,  the  government’s  strategy  does 
not  revolve  around  a  specific  bottom  line,  such  as  shareholder  wealth.  Contracting  professionals 
are  often  satisfying  multiple  entities,  such  as  regulatory  requirements  (e.g.,  the  FAR),  internal 
customers,  the  private  sector,  and  stewardship  of  taxpayer  resources.  According  to  Moore,  the 
first  element,  value, 

directs  managerial  attention  to  the  value  proposition  that  guides  the  organization. 

For  an  enterprise  to  succeed  in  producing  value,  the  leaders  of  the  enterprise  have 

to  have  a  story,  or  an  account,  of  what  value  or  purposes  that  the  organization  is 


12 


pursuing.  They  need  a  reason  for  the  organization’s  existence,  a  claim  about  the 
way  in  which  the  world  would  be  made  better  through  the  operations  of  the 
enterprise.  (Moore,  2000,  p.  197-198) 

In  essence,  value  in  a  governmental  organization  equates  to  mission. 

Moore  stated  that  legitimacy  and  support 

directs  managerial  attention  to  the  question  of  where  the  support  for  pursuing  the 
value  will  come  from.  It  is  not  enough  that  an  entrepreneurial  leader  judges  some 
purposes  to  be  valuable.  Others,  who  provide  the  necessary  financial  resources 
and  authorization,  have  to  agree  with  that  judgment.  In  government,  those  others 
include  citizens,  elected  representatives,  interest  groups,  and  the  media,  which  has 
been  called  the  ‘authorizing  environment’  of  the  organization  (Heymann,  1987; 

Moore,  1995).  (Moore,  2000,  p.  198) 

With  regard  to  the  fear  of  protest,  there  may  be  an  element  of  shame  if  a  source  selection  is 
protested,  particularly  if  there  is  a  notion  that  management  would  not  support  the  contracting 
officers  and  that  the  protest  may  reflect  poorly  on  them.  High  dollar  contracts,  in  particular,  hold 
great  interest  to  our  media  and  our  elected  officials.  A  protest  would  reflect  negatively  on  the 
contracting  official  as  well  as  the  contracting  office.  With  these  concerns  in  the  back  of  a 
contracting  officer’s  mind,  there  can  be  a  tendency  to  take  measures  in  order  to  avoid  a  protest 
that  can  sub-optimally  contribute  to  a  source  selection.  For  example,  the  officer  may  rely  too 
heavily  on  LPTA  rather  than  utilizing  a  best  value  approach.  Fear  of  a  protest  is  understandable. 
Time  has  to  be  devoted  to  address  the  requirements  of  a  protest.  Ultimately,  contracting  offices 
then  have  less  time  and  resources  to  devote  to  other  requirements.  The  needs  of  the  customer  do 
not  stop  because  of  a  protest.  This,  in  turn,  has  an  adverse  effect  on  contracting  officer’s 
legitimacy  and  decreases  the  value  of  the  contracting  element. 

Finally,  operational  capacity  focuses  attention  on  the  question  of  whether 
sufficient  know-how  and  capability  exist  to  achieve  the  desired  results.  Often,  this 
capability  lies  entirely  in  the  organization  that  the  manager  leads.  However, 
sometimes  it  lies  outside  the  organization’s  boundary,  and  the  organization  has  to 
find  ways  to  engage  capacities  beyond  its  own  to  achieve  the  desired  result  by 
creating  partnerships  of  various  kinds.  (Moore,  2000,  p.  198) 

Experience  and  workload  levels  can  often  be  elements  that  contribute  to  the  concern  of  a  protest. 
If  contracting  officers  have  adequate  experience,  their  likelihood  to  be  affected  by  this  fear  can 
diminish.  They  have  seen  many  source  selections  and  know  how  to  ensure  they  have  a  contract 
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file  that  will  stand  the  test  of  a  protest.  Conversely,  those  with  lesser  experience  and  a  fear  of 
protest  from  a  legitimacy  and  support  perspective  may  take  additional  steps  to  avoid  a  protest 
and,  in  turn,  reduce  the  value  of  the  organization  by  sub-optimally  organizing  the  contracting 
strategy.  In  contracting  there  are  often  very  busy  times,  most  notably  at  the  end  of  the  fiscal  year 
(EOFY).  As  the  organization  may  be  optimally  manned  for  most  of  the  year,  it  is  likely  they  are 
not  for  the  EOFY.  During  this  time,  requirements  are  coming  in  at  a  much  faster  rate  and  with 
less  time  to  produce  results.  Also  during  this  time,  operational  capacity  can  become  hindered  and 
lead  to  an  increased  fear  of  protest. 

Contracting  professionals  add  value  to  the  country  because  they  address  the  operational 
needs  of  our  military  and,  at  the  same  time,  provide  fairness  and  address  the  various  public 
policy  issues  that  are  required  by  law  and  regulation.  When  these  align,  customers  receive  what 
they  require  at  a  fair  and  reasonable  price,  and  this  satisfies  the  requirements  of  governing 
policies.  Through  this,  government  contracting  professionals  add  value  to  their  stakeholders. 
Contracting  officers  sometimes  take  steps  throughout  the  acquisition  process  to  avoid  a  protest, 
such  as  minimizing  discussions  or  even  employing  an  EPTA  source  selection  process  when  a  full 
trade-off  method  is  more  appropriate  (Gordon,  2013,  pp.  36-37).  When  this  occurs,  our 
contracting  system  is  not  optimizing  its  value.  When  it  treats  a  source  selection  with  a  fear  of 
protest,  the  agency’s  value  can  be  reduced. 

Contracting  officers  are  also  accountable  to  provide  fairness  to  commercial  entities  with 
which  they  contract  for  goods  and  services  (FAR  Part  1).  Often,  though,  a  fear  of  a  bid  protest 
will  result  in  multiple  award  contracts  including  more  contractors  than  would  have  been  awarded 
if  there  was  no  fear  of  a  bid  protest.  In  multiple  award  contracts  there  is  a  minimum  dollar  value 
that  a  contractor  is  obligated  to  receive.  This  results  in  increased  spending  of  taxpayer  money 
that  could  have  been  more  efficiently  spent  by  awarding  to  fewer,  more  competitive  contractors. 
This  creates  extra  work  for  the  contracting  officer,  duplicates  inventory,  can  increase 
transportation  costs,  and  results  in  non-optimal  use  of  taxpayer  money,  and  often  upset 
contractors  who  never  get  an  award  under  a  multiple  award  contract  for  which  they  believed  they 
were  competitive.  Although  a  reduced  risk  of  a  protest  is  accomplished,  the  PVF  is  disturbed. 
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and  there  is  ultimately  less  value  added  by  the  contracting  process.  What  this  does  not 
accomplish  is  a  best  option  for  the  customer  or  the  taxpayer,  nor  does  it  provide  fairness  to  the 
stronger  contractors. 

D.  STAKEHOLDER  THEORY 

The  stakeholder  theory  explains  the  people  and  entities’  interest  in  a  particular 
organization.  A  stakeholder  in  an  organization  is,  by  definition,  any  group  or  individual  who  can 
affect  or  is  affected  by  the  achievement  of  the  organization’s  objective  (Freeman,  1984). 
According  to  a  stakeholder  model  of  a  corporation  (Freeman,  2001),  stakeholders  for  a  firm  are 
owners,  suppliers,  management,  employees,  customers,  and  the  local  community.  For  this 
research,  the  focus  is  on  the  contracting  offices  as  the  employees  and  the  end-users  as  the 
customers.  Contractors  are  the  suppliers  providing  a  service  and/or  a  product,  and  the  acquisition 
leaders,  senior  contracting  officers,  staff,  and  supervisors  are  the  management.  Other 
stakeholders  in  this  research  are  congress,  legal  representation  of  the  contracting  office,  and 
taxpayers/citizens.  For  the  employees  (contracting  officers),  their  stake  in  the  firm  (contracting 
activity)  could  be  job  security,  high  wages,  and  job  satisfaction.  In  return,  employees  are 
expected  to  follow  the  instructions  of  management  most  of  the  time  (Freeman,  2001).  This 
presents  an  issue  if  fear  of  a  protest  exists  at  a  higher  principal  level.  Employees  are  expected  to 
change  their  original  acquisition  strategy  to  continue  to  be  stakeholders. 

Suppliers  (contractors)  are  vital  to  the  success  of  the  firm.  In  turn,  the  firm  is  a  customer 
of  the  supplier  (Freeman,  2001).  For  the  contracting  activity,  the  relationship  with  the  contractor 
is  vital.  The  contractor  is  a  supplier  of  proposals,  market  research  information,  innovation,  and 
competition.  What  happens  when  a  supplier  threatens  the  contracting  activity  with  a  protest  or 
there  is  a  fear  of  protest  from  a  supplier?  The  employees,  management,  and  customers  might 
change  their  original  acquisition  strategy  to  keep  the  supplier  as  a  stakeholder. 

Customers  provide  the  lifeblood  of  the  firm  (Freeman,  2001).  Without  customers,  the 
firm  would  not  exist.  All  organizations  realize  and  understand  this.  Many  organizations  treat  the 
customers  as  the  principal  and  themselves  as  an  agent  in  the  principal-agent  theory.  To  continue 
to  get  revenue  and/or  business,  the  agent  aligns  or  focuses  the  organization’s  actions  on  the 
customer’s  needs.  The  agent  needs  the  customer  as  a  stakeholder  to  continue  to  exist. 
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Management  (acquisition  leaders,  senior  contracting  officers,  and  contracting  staff  and 
supervisors)  plays  an  integral  role.  Managers  are  the  only  group  of  stakeholders  who  enter  into  a 
contractual  relationship  with  all  other  stakeholders.  Managers  are  also  the  only  group  of 
stakeholders  with  direct  control  over  the  decision-making  apparatus  of  the  firm.  Therefore,  it  is 
incumbent  upon  managers  to  make  strategic  decisions  and  allocate  resources  in  the  manner  most 
consistent  with  the  claims  of  the  other  stakeholders  groups  (Hill  &  Jones,  1992).  Management 
must  keep  the  relationships  among  stakeholders  in  balance.  When  these  relationships  become 
imbalanced,  the  survival  of  the  firm  is  in  jeopardy  (Freeman,  2001).  These  imbalanced 
relationships  among  stakeholders  and  the  threat  of  a  protest  could  be  a  factor  of  fear.  For 
example,  a  supplier  (contractor)  protests  an  award  because  the  contracting  activity  did  not 
maintain  a  good  stakeholder  relationship  with  the  supplier.  Since  there  were  no  discussions  or 
meaningful  debriefings,  the  supplier  may  protest  just  to  get  information.  Management  must 
devote  people  (employees),  time,  and  resources  to  deal  with  the  supplier’s  protest  while  still 
meeting  the  customer’s  needs.  Management  (agent)  must  satisfy  the  customer  and  the  supplier 
base,  take  care  of  its  employees,  and  be  good  stewards  of  the  taxpayers’  (principal)  money. 

Legal  in  many  contracting  activities  is  part  of  the  management  team.  Its  role  is  to  advise 
the  employees  (contracting  officers)  and  the  management  team  of  the  legal  aspect  of  acquisition 
procurement.  Legal  looks  at  the  procurement  from  a  different  prospective  to  ensure  there  are  no 
grounds  for  a  protest  from  prejudice,  errors  in  the  procurement  process,  violations  of  statutes  or 
regulations,  and  other  protestable  actions.  As  a  stakeholder,  legal  offers  advice  in  regard  to  the 
source  selection  method.  Based  on  our  interviews,  legal  has  influence  in  the  source  selection 
method. 

Besides  the  employees,  suppliers,  customers,  and  management,  there  are  other 
stakeholders  that  are  an  integral  part  of  the  contracting  process.  They  are  not  involved  in  the  day- 
to-day  operation  of  the  acquisition  process. 

Congress  is  an  important  stakeholder  in  the  acquisition  process.  Congress  exerts  authority 

over  the  contracting  process  through  the  use  of  its  constitutional  powers  (Cibinic,  Nash,  & 

Yukins,  2011,  p.  38).  The  primary  means  of  exercising  its  authority  are  through  the  enactment  of 

laws  establishing  new  programs  and  the  appropriation  of  funds  to  pay  for  these  programs 

(Cibinic  et  ah,  2011,  p.  39).  The  U.S.  Constitution,  Article  1,  Section  9,  Clause  7,  states  that  “no 
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money  shall  be  drawn  from  the  Treasury,  but  in  Consequence  of  Appropriations  made  by  law.” 
Congress  also  exercises  control  and  oversight  over  acquisition  activities  through  requirements 
and  limitations  included  in  procurement  statutes,  authorizations  acts,  appropriations  acts,  and 
other  statutes  (Cibinic  et  ah,  2011,  p.  39).  As  a  stakeholder.  Congress  authorizes  several 
administrative  or  judicial  forums  to  hear  protests  (Manuel  &  Schwartz,  2011,  p.  3).  Those 
reasons  are  to  “ensure  that  entities  doing  business  with  the  government  can  air  their  complaints 
about  governmental  contracting  processes  and  obtain  relief’  and  “enhance  the  accountability  of 
procurement  officials  and  government  agencies  by  highlighting  and  correcting  mistakes  and 
misconducf’  (Manuel  &  Schwartz,  2011,  p.  3). 

Without  a  protest  forum,  other  stakeholders  (i.e.,  suppliers  and  customers)  would  be  less 
willing  to  do  business  with  the  government,  thus  decreasing  completion,  which  could  potentially 
drive  up  the  cost  of  products  and  services  (Manuel  &  Schwartz,  201 1,  p.  3). 

Taxpayers/citizens  represent  a  wide  range  of  individuals.  They  can  be  considered  as 
suppliers  of  funds  through  government  taxation.  As  part  of  the  general  public,  taxpayers  can  also 
be  considered  customers  in  that  their  tax  dollars  are  spent  on  public  goods  and  services.  For 
example,  taxpayers  pay  taxes  to  fund  the  military.  The  military  protects  and  defends  the  general 
public  in  the  event  of  an  invasion.  Taxpayers  are  employees,  management,  suppliers.  Congress, 
legal,  stakeholders  and  so  forth  in  the  acquisition  process.  As  taxpayers  in  the  acquisition 
process,  stakeholders  have  a  vested  interest  to  see  that  their  precious  funds  were  not  wasted,  the 
procurement  was  awarded  through  the  best  source  selection  method  for  that  particular 
procurement,  and  the  acquisition  process  was  fair  to  the  public.  This  presents  a  challenge  when 
different  stakeholders  have  different  views  about  which  source  selection  method  should  be  used. 

This  research  examines  the  stakeholder  theory  costs  and  associated  problems  that  arise 
from  a  fear  of  protest.  Employees  and  management  are  primarily  concerned  about  a  protest  from 
the  supplier.  All  three  stakeholders  are  trying  to  meet  the  interest  of  the  customer. 

E.  HYPOTHESES  AND  CONCEPTUAL  MODEL 

1.  Factors  Affecting  Protest  Fear 

There  are  numerous  factors  that  affect  a  fear  of  protest.  This  study  primarily  focuses  on 

three  main  factors  contributing  to  the  research  hypotheses:  literature  reviews,  theory,  and  logic. 
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a.  Insufficient  Procurement  Administrative  Lead-Time  Planned 

Insufficient  procurement  administrative  lead-time  (PALT)  is  the  inadequate  time  allotted 
to  accomplish  an  acquisition.  Insufficient  PALT  is  often  the  result  of  fiscal  appropriation 
constraints  that  commonly  occur  toward  the  end  of  the  fiscal  year.  Expedited  requirements  and 
poor  planning  are  also  common  reasons  that  can  lead  to  insufficient  PALT.  When  sufficient  lead- 
time  is  not  allocated  to  properly  define  requirements,  evaluation  criteria,  and  instructions  to 
offerors;  train  the  technical  evaluators;  evaluate  proposals;  document  evaluations  and  trade-offs; 
and  prepare  for  and  brief  decision  makers,  protestable  errors  are  more  likely  to  occur.  Thus, 
sufficient  time  renders  the  acquisition  team  capable  of  successfully  performing  a  source 
selection.  Absent  sufficient  time,  operational  capability  is  constrained. 

Hypothesis  1:  Insufficient  PALT  has  a  direct  positive  relationship  on  the  level  of  fear  of 

protest. 

b.  Contracting  Officer’s  Competence 

Competence  is  the  contracting  officer’s  contracting  experience,  educational  background, 
and  the  professional  Defense  Acquisition  Workforce  Improvement  Act  (DAWIA)  certification 
level  the  contracting  officer  holds.  There  is  no  substitute  for  experience  and  for  the  level  of 
confidence  one  has  in  his  or  her  ability  as  a  contracting  officer.  The  more  experience  a 
contracting  officer  has,  the  less  concern  of  a  protest  there  should  be  since  the  officer  has  more 
experience  in  techniques  and  practices  to  prevent  bid  protest.  The  contracting  officer  also  has 
more  knowledge  of  how  to  defend  a  protest  if  received,  and  thus,  may  worry  about  protests  less 
because  of  the  officer’s  increased  operational  capacity  to  handle  a  protest. 

Hypothesis  2:  The  greater  a  contracting  officer’s  competence  level,  the  lower  the  level  of 

fear  of  protest. 

c.  Requirement  Criticality/Importance 

How  critical  or  important  the  requirement  is  to  the  overall  mission  of  the  customer  will 
impact  protest  concerns.  If  the  requirement  is  critical  to  the  customer’s  mission  effectiveness, 
there  could  be  a  tendency  to  use  a  procurement  method  that  does  not  delay  or  jeopardize  the 
award  of  the  contract.  This  is  because  the  value  of  a  government  agency  (e.g.,  military  unit)  is 
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determined  by  its  mission  effectiveness  (Moore,  2000).  If  the  delivery  or  performance  of  a  good 
or  service  is  delayed,  the  unit’s  mission  will  likely  be  delayed  to  some  extent,  compromising  the 
unit’s  public  value. 

Hypothesis  3:  The  level  of  requirement  criticality/importance  has  a  positive  impact  on  the 

level  of  fear  of  protest. 

d.  Protest  Risk 

Risk  is  defined  as  the  product  of  the  magnitude  of  consequences  and  the  probability  of 
their  occurrence.  The  risks  involved  with  a  protest  can  be  added  time  to  complete  the  acquisition, 
delays  to  the  mission,  which  can  directly  and  negatively  impact  the  war  fighter,  additional  effort 
and  resources  required  to  resolve  the  protest,  and  personal  shame  and  embarrassment 
experienced  by  the  acquisition  team.  Protest  may  also  be  perceived  as  bringing  negative  career 
repercussions,  such  as  preventing  promotion  or  a  direct  admonishment.  The  likelihood  of  any  of 
these  negative  consequences  depends  on  the  circumstances  unique  to  each  bid  protest. 
Nonetheless,  if  the  protest  risks  are  high,  the  consequences  could  jeopardize  the  unit’s  public 
value  (e.g.,  its  mission  effectiveness). 

Hypothesis  4:  As  protest  risk  increases,  the  level  of  fear  of  protest  increases. 

2.  Protest  Fear  and  the  Consequences  to  the  Acquisition  Strategy 

Fear  is  a  natural  reaction  to  a  threat.  As  a  natural  reaction,  fear  enables  humans  to  remove 
the  threat  deliberately  or  subconsciously.  In  this  case,  fear  of  protest  could  be  considered  a 
natural  reaction  to  the  potential  consequences  if  a  bid  protest  was  received.  Acquisition 
professionals  could  deliberately  or  subconsciously  remove  the  threat  through  their  many 
acquisition  strategy  decisions.  These  decisions  could  be  considered  actions  taken  to  prevent  a 
protest. 

There  are  numerous  consequences  to  the  acquisition  strategy  and  contract  outcomes  as  a 
result  of  fear.  This  study  focuses  on  consequences  identified  from  our  literature  review  and 
interviews  with  subject  matter  experts  and  practitioners. 
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a.  Quality  of  Technical  Evaluation  Effectiveness 

According  to  FAR  15.304(b),  evaluation  factors  and  significant  sub  factors  must  (1) 
represent  the  key  areas  of  importance  and  emphasis  to  be  considered  in  the  source  selection 
decision;  and  (2)  support  meaningful  comparison  and  discrimination  between  and  among 
competing  proposals. 

Agencies  evaluate  the  competitive  proposals  and  assess  their  relative  qualities  based  only 
on  the  factors  and  subfactors  specified  in  the  request  for  proposal  (RFP)  (Rumbaugh,  2010). 

As  part  of  the  evaluation,  proposals  are  put  in  a  competitive  range.  The  contracting 
officer  determines  which  proposals  are  within  the  competitive  range  based  on  the  evaluated  price 
and  other  evaluation  factors  included  in  the  RFP  (Rumbaugh,  2010,  p.  338).  A  low-rated 
proposal  could  potentially  be  eliminated  from  the  completive  range  if  the  proposal  does  not  meet 
the  specified  evaluation  factors  or  subfactors  in  the  RFP. 

If  the  contracting  officer  and  the  source  selection  authority  are  concerned  with  a  fear  of 
protest,  they  could  lower  the  evaluation  factors  and  subfactors  for  all  of  the  proposals.  By 
lowering  the  factors,  the  lower-rated  proposals  are  not  eliminated  from  the  competition,  which 
mitigates  the  chances  of  a  protest.  The  lower-rated  proposal  might  be  the  winning  proposal. 

Hypothesis  5:  The  level  of  fear  of  protest  has  a  negative  effect  on  the  technical  evaluation 

effectiveness. 

b.  Source  Selection  Method  Fit/ Appropriateness 

According  to  FAR  15.101-1,  a  trade-off  process  is  appropriate  when  it  may  be  in  the  best 
interest  of  the  government  to  consider  an  award  to  a  company  other  than  the  lowest  priced 
offeror  or  other  than  the  highest  technically  rated  offeror.  This  process  gives  the  source  selection 
official  very  broad  discretion  when  the  offeror  with  the  lowest  cost  to  the  government  has  not 
been  evaluated  based  on  the  non-cost  factors  (Cibinic  et  ah,  2011,  p.  678). 

According  to  FAR  15.101-2,  the  LPTA  source  selection  process  is  appropriate  when  best 
value  is  expected  to  result  from  selection  of  the  technically  acceptable  proposal  with  the  lowest 
evaluated  price.  There  are  many  reasons  why  a  contracting  officer  might  opt  for  the  LPTA.  One 
major  benefit  of  this  strategy  is  that  the  agency  can  greatly  shorten  the  evaluation  process 
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because  once  the  low  price  proposal  has  been  found  to  be  technically  acceptable;  there  is  no  need 
to  evaluate  the  acceptability  of  any  other  proposals.  (Cibinic  et  ah,  2011,  p.  680). 

The  source  selection  appropriateness  depends  on  the  procurement  requirement. 
According  to  one  interviewee, 

Again,  because  my  philosophy  has  always  been  what  is  the  best  way  to  make  sure 
the  customer  gets  what  they  want.  Is  it  going  to  be  something  that  either  yes,  they 
can  offer  or  provide  the  product  or  service;  yes  or  no,  so  it  is  a  slam  dunk.  Low 
price,  technically  acceptable — or  do  you  want  to  have  the  flexibility  of  best  value 
where  it  is  something  that  you  want  to  encourage  innovation  or  design  or 
whatever  else? 

According  to  FAR  Part  6.3,  there  are  many  reasons  to  use  a  method  other  than  full  and 
open  competition.  Based  on  the  situation,  a  contracting  officer  might  use  any  of  those  reasons 
with  a  valid  justification  and  approval.  If  a  contracting  officer  is  concerned  about  a  protest,  he  or 
she  could  structure,  with  justification,  the  requirement  for  a  sole  source  acquisition.  Having  an 
approved  justification  restricts  competition  to  a  sole  source.  This  prevents  other  companies  from 
protesting.  The  use  of  sole  source  works  well  in  situations  where  there  is  an  emergency,  the 
mission  is  in  jeopardy,  and/or  time  is  very  critical. 

A  contracting  officer  may  not  offer  discussions  because  having  communications  could 
potentially  cause  a  protest.  One  of  the  reasons  businesses  file  protests  is  because  agencies  do  not 
adequately  debrief  losing  companies.  Those  losing  companies  may  file  a  protest  just  to  get 
information  on  why  they  lost  the  competition  (Schwartz  et  ah,  2013). 

A  contracting  officer  could  also  be  afraid  to  talk  to  businesses.  Every  word  the 
contracting  officer  says  could  potentially  be  used  in  the  protest  case.  During  a  CSIS  conference 
with  Kendall,  the  moderator  described  this  as  “the  fear  of  conducting  discussions  because 
contracting  officers  get  it  into  their  head  that  if  they  conduct  discussions,  they’re  going  to  end  up 
with  a  bid  protest,  and  it’s  just  going  to  take  too  long,  and  they’ll  get  dinged”  (Kendall,  2012). 

According  to  Rumbaugh,  if  the  agency  has  already  received  good  proposals,  then  going 
into  discussions  is  not  likely  to  provide  more  value.  It’s  “not  a  good  practice  to  conduct 
discussions  to  lower  offerors’  prices  when  initial  offers  are  fair  and  reasonable”  (Rumbaugh, 
2010,  p.  326). 
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Hypothesis  6:  There  is  a  direct  positive  relationship  between  fear  of  protest  and  the 

Source  Selection  method  fit/  appropriateness. 

c.  Contractor  Performance 

According  to  Gansler,  Lucyshyn,  and  Arendt  (2009),  competition  is  important  for  the 
DOD  because  it  can  aid  in  increasing  economic  efficiency,  innovation,  quality,  and  performance. 
Competition  can  aid  in  economic  efficiency  (for  both  systems  and  services)  by  forcing  firms  to 
keep  prices  as  low  as  possible  in  hopes  of  winning  a  contract.  Likewise,  increased  innovation 
results  from  competitive  pressures  as  firms  seek  new  solutions  to  DOD  problems  through 
research  and  development  while  trying  to  differentiate  themselves  from  competitors. 
Competition  also  impacts  quality  as  firms  attempt  to  improve  the  quality  of  a  product  over  time, 
which  has  the  potential  to  drive  lower-quality  firms  out  of  the  market  altogether.  Finally, 
competition  can  also  directly  impact  performance  improvements  for  both  weapons  systems  and 
services.  (Gansler  et  ah,  2009,  p.  14) 

The  use  of  LPTA  only  requires  the  lowest  price  among  offerors  that  meet  the  acceptable 
technical  requirements.  In  some  instances  the  quality  of  technical  evaluations  are  degraded 
because  of  the  check  in  the  box  mentality  that  the  LPTA  approach  brings.  Contractors  have  no 
incentive  to  improve  on  quality,  economic  efficiency,  innovation,  and  performance.  As  a  result, 
LPTA  contracts  might  result  in  lower  standards  and  performance.  According  to  an  article  in 
National  Defense  Magazine,  “He  [president  of  a  satellite  company]  pointed  to  one  study  by  the 
research  firm  Market  Connections  in  October  that  said  LPTA  contracts  may  result  in  standards 
and  performance  being  lowered,  and  less  than  optimal  prices  for  product  development” 
(Magnuson,  2014).  These  lower  standards  and  performance  cost  the  government  in  schedule 
delays,  cost  overruns,  poor  contractor  performance,  rework,  and  ultimately  unsatisfied 
customers.  In  cases  where  the  requirement  requires  innovation  or  complex,  and/or  risk-taking 
requirements  on  both  the  government  and  contractor,  LPTA  might  not  be  the  appropriate  best 
value  method.  For  those  requirements,  the  government  is  looking  for  the  best  overall  proposal 
value.  If  LPTA  was  used  for  those  innovations  or  complex  or  risky  requirements,  the  contractor 
could  be  unable  to  complete  the  task  because  it  truly  did  not  understand  the  requirement  reflected 
in  the  lowest  price  and  minimum  acceptable  technical  approach.  The  government  could  have  to 
terminate  the  contract  for  default  and  then  re-solicit  and  re-compete  the  requirement,  which  cost 
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funds  and  resources.  According  to  Lohfeld,  “When  the  government  applies  the  strategy  [LTPA] 
to  unsuitable  procurements,  both  the  government  and  the  bidders  lose”  (2012  p.  1). 

The  use  of  sole  source  versus  competition  does  not  incentivize  the  contractor  to  improve 
on  quality,  economic  efficiency,  innovation  and  performance.  Sole  source  procurements  create  a 
monopoly.  The  monopoly  can  set  its  prices  to  yield  the  maximum  profit.  The  monopoly  has  no 
incentives  to  reduce  price  or  improve  on  the  product’s  or  service’s  capabilities.  The  government 
could  potentially  pay  for  the  highest  price  for  the  least  amount  of  innovation,  performance,  and 
quality.  The  trade-off  to  the  sole  source  versus  competition  is  the  unusual  and  compelling 
situation  in  which  time  is  of  the  essence. 

Contractor  performance  is  the  measure  of  how  well  the  contractor  performed  the  contract 
based  on  the  fit  of  the  source  selection  and  the  quality  of  the  technical  evaluation.  Specifically,  it 
measures  how  well  the  contractor  met  the  requirements  in  terms  of  schedule,  cost,  and 
performance,  as  well  as  the  terms  and  conditions  set  forth  in  the  contract.  Contractor 
performance  measures  how  well  the  contractor  met  the  customers’  expectations. 

Hypothesis  7:  There  is  a  positive  correlation  between  the  fit/appropriateness  of  a  source 

selection  and  contractor  performance. 

Hypothesis  8:  There  is  a  positive  relationship  between  technical  evaluation  effectiveness 

and  contractor  performance. 

d.  Buyer  Satisfaction 

Customer  satisfaction  is  important.  According  to  the  stakeholder  theory,  customers  are 
the  lifeblood  of  any  organization  (firm).  Ensuring  their  satisfaction  is  critical  to  an  organization’s 
existence.  The  role  of  management  within  the  organization  is  to  ensure  that  the  appreciable 
measures  (e.g.,  customer  service,  highest  quality  products  and  services,  and  professionalism)  are 
in  place  to  ensure  its  customers  keep  coming  back.  For  the  contracting  activity,  the  customer 
cannot  set  up  or  bind  the  government  into  contracts.  The  customer  does  not  know  the  intricate 
procurement  system  of  regulations  and  statutes.  It  is  important  for  the  agency  to  act  on  the 
customer’s  behalf.  The  agency  must  award  contracts  to  suppliers  (contractors)  to  provide  a 
product  or  service  to  the  customer.  The  suppliers  chosen  to  provide  or  perform  the  service  play 
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an  important  role  in  customer  satisfaction.  The  agency’s  commitment  to  the  customer  is 
displayed  by  responsiveness  and  resolution  of  customer  concerns,  problems,  and  complaints 
(Anderson,  Swaminathan,  &  Mehta,  2013). 

If  the  contracting  officer  (employee)  has  a  concern  about  the  customer’s  concerns, 
problems,  and  complaints,  the  contracting  officer  could  change  the  acquisition  plan  to  using 
trade-off  or  LPTA.  The  contracting  officer  could  use  LPTA  instead  of  trade-off  to  keep  a  certain 
contractor  in  the  competition  to  satisfy  the  customer.  If  the  contracting  officer  is  concerned  about 
a  protest,  he  or  she  could  use  LPTA. 

The  use  of  LPTA  could  potentially  cause  degradation  in  the  customer’s  satisfaction.  As 

companies  reduce  their  costs  to  meet  the  terms  and  conditions  of  the  contract,  they  cut  key 

elements  (e.g.,  people,  level  of  service,  material  quality)  of  their  productions.  Cutting  key 

elements  decreases  the  performance  and  increases  the  cost  and  the  amount  of  time.  This 

degradation  causes  lower  buyer  satisfaction,  as  the  customer  must  add  more  funds  and  time  to 

the  procurement  while  losing  out  on  the  original  performance  parameters  set. 

Hypothesis  9:  There  is  a  direct  positive  relationship  between  contractor’s  performance 
based  on  the  source  selection  method  used  and  the  buyer ’s  satisfaction. 

e.  PALT  Actual 

PALT  is  one  metric  that  can  increase  if  fear  causes  members  of  the  source  selection  team 
to  re-evaluate  themselves.  Naturally,  as  the  concern  over  a  protest  grows,  acquisition  teams  take 
added  measures  to  prevent  them.  This  will  increase  the  iterations  of  documents  that  are  generated 
and  will  ultimately  prolong  the  review  process.  These  added  measures  consume  time  during  the 
source  selection.  Therefore,  it  is  hypothesized  that: 

Hypothesis  10:  There  is  a  direct  positive  relationship  between  fear  of  protest  and  the 

actual  PALT. 

f  Resources 

The  composition  of  the  SST  is  customized  for  each  acquisition.  The  structure  of  the  SST 
depends  on  the  nature  of  the  acquisition  and  the  agency’s  regulations  (Rumbaugh,  2010,  p.  52). 
The  source  selection  council  (SSC)  is  another  group  that  is  added  to  the  SST.  The  SSC  function 
is  to  be  advisory  and  an  independent  reviewer  of  high  dollar  or  high  visible  acquisitions.  When 
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the  SSC  is  not  formed,  other  government  advisors  (e.g.,  legal)  can  function  and  provide  their 
expertise  (Office  of  the  Under  Secretary  of  Defense  of  Acquisition,  Technology,  and  Logistics 
(OUSD  [AT&L)),  2011,p.  7) 

The  typical  source  selection  board  includes  no  more  than  three  to  five  members. 
However,  the  more  significant  procurement  is  to  an  agency,  the  larger  the  board  is  likely  to  be 
(Edwards,  2006,  p.  69).  To  avoid  a  protest,  an  agency  might  add  more  members  to  the  source 
selection  boards. 

Hypothesis  11:  There  is  a  direct  positive  relationship  between  fear  of  protest  and  the 

number  of  Human  Resources  assigned  to  the  source  selection  team. 

g.  Transaction  Costs 

The  DOD  has  experienced  a  significant  increase  in  the  number  of  competitive  source 
selection  decisions  that  are  protested  by  industry.  Protests  are  extremely  detrimental  to  the 
warfighter  and  the  taxpayer.  These  protest  actions  consume  vast  amounts  of  the  time  of 
acquisition,  legal,  and  requirements  team  members,  and  delay  program  initiation  and  the  delivery 
of  capability  (Schwartz  et  ah,  2009,  p.  13). 

The  transaction  cost  focuses  on  the  organization  of  transactions  that  occur  whenever  a 
good  or  service  is  transferred  from  a  provider  to  a  user  across  a  technologically  separable 
interface.  When  transactions  occur  within  an  organization,  the  transaction  costs  can  include 
managing  and  monitoring  personnel  and  procuring  inputs  and  capital  equipment  (Pint  & 
Baldwin,  1997). 

In  this  research,  the  transaction  costs  reflect  the  monetary  cost  of  resources  devoted  to 
preventing  a  protest.  This  includes  the  different  acquisition  professions  (contracting  officer, 
contracting  specialist,  technical  evaluator,  legal,  cost/price  analyst,  past  performance  team, 
program  manager.  Small  Business  Association  [SBA]  representative,  and  consultant),  their  labor 
rates,  and  the  amount  of  their  time.  This  transaction  cost  could  be  considered  an  opportunity  cost 
of  resources  not  devoted  to  other  requirements  and  contracts  in  the  queue. 

Hypothesis  12:  There  is  a  direct  positive  relationship  between  fear  of  protest  and 

transaction  costs. 
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h.  Number  of  Awarded  Contracts 

Multiple  award  contracts  (MACs)  are  solicitations  that  are  broken  into  several  contracts 
for  different  companies.  This  includes  task  orders  and  indefinite  delivery /indefinite  quantity 
(IDIQ).  A  contracting  officer  might  use  MACs  because  they  offer  many  advantages.  According 
to  the  Office  of  Management  and  Budget, 

In  order  for  agencies  to  take  continuous  advantage  of  the  benefits  of  competition 
after  contract  award,  Federal  Acquisition  Streamlining  Act  (FASA)  provides  that 
agencies  may  make  multiple  awards  of  task  and  delivery  order  contracts  for  the 
same  or  similar  supplies  or  services  (and  from  the  same  solicitation)  to  two  or 
more  sources.  The  use  of  multiple  award  contracts  allows  agencies  to  take 
continuous  advantage  of  the  competitive  forces  of  the  commercial  marketplace 
which  will  result  in  lower  prices,  better  quality,  and  reduced  time  from 
requirements  identification  to  award,  and  improved  contractor  performance  in 
satisfying  customer  requirements.  (Office  of  Federal  Procurement  Policy  [OFPP], 

1997) 

One  of  the  other  advantages  of  using  MACs  is  to  reduce  the  likelihood  of  a  protest  by  awarding 
contracts  to  multiple  companies. 

Under  FAR  16.505  10  (i)  (a),  no  protest  under  FAR  33.1  is  authorized  in  connection  with 
the  issuance  or  proposed  issuance  of  an  order  under  a  task-order  contract  or  delivery -order 
contract,  except  for  “a  protest  on  the  ground  that  the  order  increases  the  scope,  period,  maximum 
value  of  the  contract.” 

For  the  Navy’s  LCS  program,  the  Navy  wanted  to  award  two  contracts  to  two  competing 
shipyards.  This  increased  competition,  lowered  the  cost,  and  prevented  a  protest.  A  December 
13,  2010  press  report  on  the  LCS  program  stated,  “One  high-level  Navy  source  recently  said  that 
without  the  dual-ship  approach,  ‘there  is  100  percent  chance  of  a  protest’”  (O’Rourke,  2014,  p. 
68).  As  a  result  of  this  fear,  the  Navy  changed  its  acquisition  plan  from  a  down  select  to  a  dual¬ 
award  strategy. 

Hypothesis  13:  The  higher  the  level  of  fear  of  protest,  the  greater  of  the  number  of 
awarded  contracts. 

i.  Contracting  Officer  Authority 

In  some  agencies,  legal  employees  are  considered  advisors.  According  to  Rumbaugh 

(2010,  p.  270),  advisors  are  subject  matter  experts  who  provide  technical  expertise  and  advice  to 
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the  evaluators.  Advisors  may  read  the  proposal,  document  their  findings,  and  make 
recommendations  to  the  evaluators.  Advisors  offer  specific  input  in  their  area  of  expertise  and 
may  only  be  asked  to  evaluate  a  limited  subset  of  evaluation  factors. 

As  advisors,  legal  is  part  of  the  acquisition  process.  Legal  is  invited  to  the  technical 
evaluations.  In  some  agencies,  it  is  an  internal  policy  to  have  legal  review  proposals  before  they 
go  forward.  If  there  is  a  protest,  legal  must  defend  the  agency’s  actions  against  the  GAO  and/or 
the  protester’s  legal  team.  Legal  must  devote  hours  and  resources  away  from  other  requirements. 
In  many  cases,  its  focus  is  on  the  protest.  Legal  understands  the  importance  of  the  protest  and  the 
effects  it  has  on  the  agency  and  the  customer.  Not  only  is  legal’s  time  devoted  to  the  protest,  but 
the  contracting  officers’  and  evaluators’  time  is  also  a  factor  when  justifying  its  actions  to  the 
Government  Affairs  Office,  protester’s  legal  time,  or  the  parent  agency.  With  a  lot  of  legal 
consequences,  legal  is  overly  conservative  of  preventing  a  protest.  Since  legal  plays  an  important 
role  in  the  acquisition  process.  Contracting  officers  rely  heavily  on  its  opinion  and 
recommendations.  This  statement  is  supported  by  the  feedback  received  during  our  interviews. 
According  to  one  interviewee,  “They  [legal]  are  only  advisory,  but  we  rely  very  heavily  on  their 
opinion.”  Another  interviewee  shared  the  same  concern: 

Part  of  legal’s  opinion  may  sometimes  tend  to  be  narrow-minded.  Sometimes  they 
tend  to  not  consider  the  real  world.  How  should  I  put  that?  Hmm.  They  might  see 
things  their  way  and  not  really  how  it  works.  We  almost  never  move  forward 
unless  they  [legal]  give  us  their  okay.  It  would  be  very,  very  hard — very 
challenging. 

Contracting  officers  are  the  only  ones  that  can  legally  bind  the  government  in  a  contract. 
They  have  the  final  authority.  Legal,  in  its  advisory  role,  influences  the  contracting  officer’s 
authority  through  its  opinion  and  recommendations. 

Hypothesis  14:  There  is  a  direct  negative  relationship  between  fear  of  protest  and  the 
contracting  officer’s  authority. 

F.  CONCLUSION 

A  total  of  14  relationships  were  hypothesized  based  on  the  relevant  literature.  A  depiction 

of  the  conceptual  model  can  be  found  in  Figure  1.  A  summary  of  the  hypothesis  can  be  found  in 

Table  2.  Chapter  III  presents  the  research  and  statistical  methods  that  are  employed,  along  with 

the  measurement  scales  that  are  used  to  measure  the  relationships  in  the  conceptual  model. 
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Table  2.  Summary  Table  of  Hypothesis 

Notation _ Hypothesis _ 

HI  Insufficient  PALT  has  a  direct  positive  relationship  on  the  level  of  fear  of 

protest. 

H2  The  greater  a  contracting  officer’s  competence  level,  the  lower  the  level  of 

fear  of  protest. 

H3  The  level  of  requirement  criticality /importance  has  a  positive  impact  on  the 

level  of  fear  of  protest. 

H4  As  protest  risk  increases,  the  level  of  fear  of  protest  increases. 

H5  The  level  of  fear  of  protest  has  a  negative  effect  on  the  technical  evaluation 

effectiveness. 

H6  There  is  a  direct  positive  relationship  between  fear  of  protest  and  the  Source 

Selection  method  fit/  appropriateness. 

H7  There  is  a  positive  correlation  between  the  fit/appropriateness  of  a  source 

selection  and  contractor  performance. 

H8  There  is  a  positive  relationship  between  technical  evaluation  effectiveness 

and  contractor  performance. 

H9  There  is  a  direct  positive  relationship  between  contractor’s  performance 

based  on  the  source  selection  method  used  and  the  Buyer’s  satisfaction. 

HIO  There  is  a  direct  positive  relationship  between  fear  of  protest  and  the  actual 

PALT. 

Hll  There  is  a  direct  positive  relationship  between  fear  of  protest  and  the 

number  of  Human  Resources  assigned  to  the  source  selection  team. 

HI 2  There  is  a  direct  positive  relationship  between  fear  of  protest  and  transaction 

costs. 

HI 3  The  higher  the  level  of  fear  of  protest,  the  greater  of  the  number  of  awarded 

contracts. 

H14  There  is  a  direct  negative  relationship  between  fear  of  protest  and  the 

contracting  officer’s  authority. 
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Figure  1 .  Conceptual  Model 
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III.  METHODOLOGY 


A.  INTRODUCTION 

This  chapter  begins  by  discussing  the  interview  construction  and  rationalizing  the  sample 
by  discussing  the  method  used  to  sample  the  population  of  contracting  professionals.  Next,  the 
survey  design  and  construct  measurements  are  presented.  Finally,  the  design  and  validation  of 
the  conceptual  model  for  analyzing  the  antecedents  and  consequences  of  protest  fear  along  with 
the  statistical  methodology  for  analyzing  the  relationships  of  the  model  are  discussed. 

B.  INTERVIEWS 

The  next  step  in  the  validation  process  was  to  interview  subject  matter  experts.  The 
purpose  was  to  validate  the  hypothesized  model  and  to  develop  and  validate  the  measures  of 
constructs  that  did  not  have  existing  measurement  scales.  As  such,  the  interviews  provided 
content  validity  and  construct  validity.  Navy  contracting  officers  at  the  Commander  Naval  Space 
and  Warfare  Headquarters  and  at  the  Navy  Fleet  Logistics  Center  San  Diego  were  chosen  for 
interviews  because  of  (a)  the  convenience  of  travel,  (b)  a  willingness  to  support  the  research,  and 
(c)  the  availability  of  a  wide  variety  of  contract  types  and  contracted  goods  and  services  for  wide 
generalizability  (e.g.,  external  validity).  A  series  of  questions  (see  Table  3)  was  asked  to  each 
participant.  All  responses  were  unrehearsed  and  not  coerced. 


Table  3.  Interview  Questions 

_ Interview  question _ 

1  How  important  to  you  is  avoiding  a  bid  protest? 

2  Why  is  avoiding  a  bid  protest  important? 

3  What  are  the  negative  consequences  of  a  bid  protest? 

4  Are  there  any  positive  outcomes  of  receiving  a  bid  protest? 

5  Are  some  members  of  the  source  selection  team  more  fixated  on  avoiding  a 
bid  protest  than  others?  Who?  Why? 

6  If  there  were  no  ability  to  protest,  would  you  have  done  anything  differently 
in  the  past  on  a  source  selection  (e.g.,  acquisition  strategy  elements)? 

7  Do  you  believe  that  source  selection  teams  alter  acquisition  strategies  in 
order  to  avoid  bid  protests?  What  are  the  outcomes  of  these  alterations? 

8  What  extraordinary  measures  have  you  observed  or  heard  of  that  source 

_ selection  teams  have  taken  to  avoid  a  bid  protest? _ 
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A  total  of  18  individuals  were  interviewed  over  two  days  between  the  two  commands. 
Demographics  of  each  respondent  can  be  found  in  Table  4.  Each  interview  was  recorded  and 
transcribed.  The  average  interview  lasted  26  minutes.  The  interviews  resulted  in  eight  hours  and 
three  minutes  of  recordings,  which  transcribed  into  229  pages. 

Interviewees  were  given  a  copy  of  the  conceptual  model  during  the  interview  and  asked 
whether  they  agreed  with  the  independent  variables  being  used.  They  were  also  asked  if  they 
would  add  any  or  take  any  away.  One  of  the  respondents,  a  supervisory  contracting  officer  with 
29  years  of  experience,  stated,  “Okay.  This  is  good.  I  don’t  see  anything  that  I  need  to  add.” 
Another  contracting  officer  with  18  years  of  experience  who  had  recently  dealt  with  a  pre-award 
protest  stated,  “I  think  this  is  a  great  research  that  you  are  doing  because  this  is  a  bigger  and 
bigger  issue.  I  think  you  are  right  on.”  Other  statements  that  validated  the  model  were,  “I  think  I 
like  the  model.  For  the  most  part  it  says  everything.” 
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Table  4.  Interview  Demographics 


Rank/Grade 

Gender 

Duty  Title 

Yrs. 

Exp 

# 

Source 

Sel. 

(within 

2  yrs) 

#Bid 
Protest 
(within  2 
yrs) 

Contract 

Types 

GS-14 

F 

Supv.  Contracting  officer 

29 

12 

0 

CPFF,  T&M, 
FFP 

GS-13 

M 

Contracting  officer 

26 

9 

3 

CPFF,  T&M, 
FFP 

GS-13 

M 

Contracting  officer 

18 

6 

1 

Various 

GS-14 

M 

Supv.  Contracting  officer 

36 

1 

0 

Various 

GS-13 

F 

Contracting  officer 

6 

4 

0 

CPFF,  T&M, 
FFP 

GS-14 

F 

Supv.  Contracting  officer 

32 

3 

0 

Various 

GS-12 

M 

Contract  Specialist 

4 

0 

0 

Various 

GS-13 

M 

Contracting  officer 

6 

6 

0 

Various 

GS-12 

M 

Contract  Specialist 

7 

4 

2 

FFP,  CPFF, 
CPAF 

GS  13 

M 

Contracting  officer 

21 

1 

0 

FFP,  Cost 

GS  13 

F 

Contracting  officer 

30 

4 

0 

FFP,  Cost 

GS  13 

M 

Contracting  officer 

22 

5 

0 

CPFF;  CPIF; 
FFP 

GS  13 

F 

Contracting  officer 

15 

5 

1 

CPFF;  CPAF 

GS  13 

M 

Contracting  officer 

26 

20 

1 

FFP;  CPFF 

C.  POPULATION  AND  SAMPLE 


After  the  interview  testimonies  confirmed  the  conceptual  model,  a  survey  to  collect  the 
quantitative  data  was  developed.  The  population  for  this  study  consisted  of  only  the  continental 
United  States  (CONUS)  and  outside  the  continental  United  States  (OCONUS)  U.S.  Navy  civilian 
and  military  contracting.  The  research  was  limited  to  the  Navy  because  of  time  and  sponsorship 
constraints.  The  sample  was  constrained  to  contracting  officers  who  had  executed  a  FAR  Part 
15-based  formal  source  selection  with  a  dollar  amount  of  greater  than  $150,000.  The  rationale 
was  to  remove  contracting  officers  who  primarily  execute  acquisitions  below  the  simplified 
acquisition  threshold  and  who  conduct  non-competitive  procurements.  Several  constraints 
existed  for  surveying  this  population.  A  list  of  e-mail  addresses  was  generated  using  data 
extracted  from  the  Federal  Procurement  Data  System-Next  Generation  (FPDS-NG)  database  to 
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encompass  all  transactions  that  fit  the  criteria  previously  stated.  Because  of  the  costs  required  to 
mail  a  sufficient  quantity  of  surveys  to  contracting  personnel  and  the  resources  required  to 
manually  input  responses  into  a  dataset,  a  web-based  survey  was  deployed  to  each  individual  e- 
mail  address  generated  from  FPDS-NG  data. 

The  unit  of  analysis  for  this  survey  was  a  source  selection.  Since  nearly  all  bid  protests 
stem  from  a  protestable  action  (e.g.,  a  proposal  rating,  rating  justification,  or  basis  of  a  trade-off 
analysis)  associated  with  a  source  selection,  this  is  the  proper  unit  of  analysis  for  the  study. 
Respondents  were  instructed  to  answer  the  survey  questions  using  their  experience  from  their 
most  recently  completed  FAR  Part  15  source  selection.  The  most  recent  source  selection  was 
required  to  serve  as  the  basis  of  reference  in  order  to  prevent  respondents’  self-selection  bias. 

D.  SURVEY  DESIGN  AND  CONSTRUCT  MEASUREMENT 

There  were  115  questions  used  to  measure  the  16  variables  in  the  conceptual  model.  The 
interview  informants  reviewed  the  measures  of  new  constructs  and  found  their  content  to  capture 
the  full  meaning  of  the  constructs  being  measured.  All  scales  measuring  latent  constructs  used  a 
Likert-type  scale.  Questions  collecting  continuous  data,  such  as  dollar  value,  planned  PALT, 
actual  PALT,  iterations  of  source  selection  documents,  and  quantity  of  personnel  assigned  did 
not  use  Likert-type  scales. 

Fear  of  protest  is  a  term  used  for  this  research  to  identify  the  level  of  apprehension  a 
contracting  professional  has  about  receiving  a  bid  protest.  Since  the  fear  of  protest  construct  was 
substantiated  during  the  literature  review  and  through  qualitative  interviews,  six  questions  were 
constructed  to  measure  the  level  of  fear  of  protest  among  the  respondents  to  the  survey,  shown  in 
Table  5.  No  previously-validated  scales  were  available  to  measure  the  fear  of  protest.  One 
variable  from  this  construct  was  discarded  as  a  result  of  factor  analysis. 
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Table  5.  Construct  Measurement  of  Fear  of  Protest 


Scale  Item 

Survey  Question 

FEAR** 

At  some  point  during  the  development  of  the  acquisition  strategy  or  the  source 
selection  process,  I  worried  about  receiving  a  bid  protest. 

FEAR^* 

I  was  concerned  that  the  contract  award  would  be  protested. 

EEAR^* 

I  was  anxious  to  get  beyond  the  10-day  point  after  contract  award  (or 
debriefings)  to  determine  whether  or  not  the  contract  would  be  protested. 

EEAR^*# 

Receiving  a  bid  protest  would  have  been  among  the  worst  things  that  could 
happen. 

EEAR^** 

During  the  development  of  the  acquisition  strategy  and  throughout  proposal 
evaluation,  to  what  extent  were  you  concerned  that  an  offeror  might  protest  the 
contract  award? 

EEAR®** 

During  the  development  of  the  acquisition  strategy  and  throughout  proposal 
evaluation,  to  what  extent  was  at  least  one  other  member  of  the  source  selection 
team  concerned  that  an  offeror  might  protest  the  contract  award? 

*  anchors  of  strongly  agree  and  strongly  disagree 
**  anchors  of  not  at  all  concerned  and  extremely  concerned 
#  item  was  discarded  due  to  Exploratory  Factor  Analysis 


The  contracting  officer  authority  construct  described  how  empowered  the  contracting 
officer  is  to  make  final  decisions  during  the  source  selection  process.  The  contracting  officer 
authority  factor  was  validated  through  qualitative  interviews.  Six  questions  were  constructed  to 
measure  the  contracting  officer’s  authority  to  make  decisions,  which  are  shown  in  table  6.  There 
were  no  previously-validated  scales  available  for  this  construct.  The  construct  consisted  of  six 
questions  using  a  seven-point  Likert-type  scale  with  anchors  of  strongly  agree  and  strongly 
disagree.  One  variable  from  this  construct  was  discarded  as  a  result  of  factor  analysis. 
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Table  6.  Construct  Measurement  of  Contracting  Officer  Authority 


Scale  Item 

Survey  Question 

co_auth‘ 

I  was  empowered  to  make  required  decisions  throughout  the  source  selection. 

CO_AUTH^ 

I  was  trusted  that  the  decisions  I  made  throughout  the  source  selection  would 
be  appropriate. 

CO_AUTH^ 

My  management  supported  me  on  the  decisions  I  made  during  the  source 
selection. 

CO_AUTH^ 

If  I  disagreed  with  an  aspect  of  a  legal  opinion/review,  I  had  the  latitude  to 
deviate  from  it. 

CO_AUTH^#* 

I  had  to  change  documents  generated  during  the  source  selection  to 
correspond  with  reviewers. 

CO.AUTH'’* 

I  might  as  well  not  have  a  warrant  since  my  decisions  were  overridden  by 
reviewers. 

#  item  was  discarded  due  to  Exploratory  Factor  Analysis 

*  item  was  reverse  coded 


The  antecedent  insufficient  PALT  was  validated  by  qualitative  interviews  and  vetting  by 
contracting  practitioners.  Three  questions,  shown  in  Table  7,  were  constructed  to  measure  the 
sense  of  urgency  created  by  insufficient  PALT  planned  in  the  milestones  and  allocated  by  the 
acquisition  team  and  its  managers  to  conduct  the  source  selection.  The  scales  were  developed 
from  a  study  on  services  sourcing  performance  (Muir,  2010).  The  variable  consisted  of  three 
questions  using  a  seven-point  Likert-type  scale  with  anchors  of  strongly  agree  and  strongly 
disagree. 


Table  7.  Construct  Measurement  of  Insufficient  PALT  Planned 

Scale  Item _ Survey  Question _ 

PALT_P**  The  milestones  for  awarding  this  contract  were  too  aggressive. 

PALT_P  I  was  not  rushed  to  award  this  contract. 

■j 

PALT_P _ I  had  sufficient  time  to  get  this  contract  awarded. _ 

*  item  was  reverse  coded 
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The  technical  evaluation  effectiveness  construct  was  also  validated  by  qualitative 
interviews.  Six  questions  were  constructed,  shown  in  Table  8,  to  measure  the  technical 
evaluators’  level  of  confidence  associated  with  the  quality  of  the  technical  evaluations  of 
offerors’  proposals.  There  were  no  previously- validated  scales  available  for  this  factor.  The 
variable  consisted  of  six  questions  using  a  seven-point  Likert-type  scale  with  anchors  of  strongly 
agree  and  strongly  disagree.  Three  variables  from  this  construct  were  discarded  as  a  result  of 
factor  analysis. 


Table  8.  Construct  Measurement  of  Technical  Evaluation  Effectiveness 

Scale  Item _ Survey  Question _ 

TEE*#*  At  least  once,  a  technical  evaluator  was  required  to  change  the  wording  of  his 

or  her  technical  evaluations. 

TEE  *  At  least  one  technical  evaluator  expressed  concern  about  not  being  able  to 

say  what  needs  to  be  said  in  the  technical  evaluation. 

TEE  *  At  least  one  technical  evaluator  was  concerned  that  the  constraints  imposed 

on  his  or  her  evaluations  impeded  the  evaluator’s  ability  to  write  a 
meaningful  evaluation. 

tee"*#*  The  technical  evaluators  believed  that  the  quality  of  their  evaluations  could 

not  have  been  better. 

TEE^#*  If  there  were  no  Eederal  Acquisition  Regulations,  no  source  selection  policy, 

and  no  threat  of  a  bid  protest,  the  quality  of  the  technical  evaluations  would 
have  been  the  same. 

TEE^*  Upon  evaluation  of  proposals,  at  least  one  technical  evaluator  expressed  a 

_ need  to  change  at  least  one  evaluation  criterion  or  its  definition. _ 

#  item  was  discarded  due  to  Exploratory  Factor  Analysis 

*  item  was  reverse  coded 

Requirement  criticality/importance  is  the  level  of  significance  of  a  particular  acquisition 
at  the  macro  level.  The  requirement  criticality/importance  factor  was  validated  by  qualitative 
interviews.  Table  9  shows  the  five  questions  that  were  used  to  measure  the  acquisition’s 
importance  with  regard  to  the  overall  mission  of  the  agency.  The  questions  were  formulated  from 
studies  conducted  on  procurements  in  business-to-business  markets  (Schoenherr  &  Mabert, 
2011).  The  variable  used  a  seven -point  Eikert-type  scale  with  anchors  of  strongly  agree  and 
strongly  disagree.  Two  variables  from  this  construct  were  discarded  as  a  result  of  factor  analysis. 


37 


Table  9.  Construct  Measurement  of  Requirement  Criticality /Importance 


Scale  Item 

Survey  Question 

RCI'# 

This  requirement  was  important  for  the  good  operation  of  our  customer’s 
organization. 

RCI^ 

This  requirement  supported  a  core  competency  of  our  customer’s 
organization. 

RCI^ 

Compared  to  other  purchases  for  this  customer,  this  requirement  was 
important. 

RCf* 

An  unsuccessful  outcome  of  the  RFP  would  have  had  only  minor 
consequences  to  our  customer 

RCI^# 

RCI*^ 

As  a  portion  of  the  customer’s  total  annual  spending  amount,  the  dollar  value 
of  this  requirement  was  high. 

As  a  portion  of  the  customer’s  total  annual  spending  amount,  the  dollar  value 
of  this  requirement  was  high. 

#  item  was  discarded  due  to  Exploratory  Factor  Analysis 

*  item  was  reverse  coded 


Buyer  satisfaction  represents  the  level  of  contentment  the  agency  has  with  the 
contractor’s  performance  that  won  the  source  selection  in  question.  The  buyer  satisfaction  factor 
was  validated  by  qualitative  interviews.  Table  10  shows  the  five  questions  that  were  constructed 
to  measure  the  level  of  satisfaction  that  resulted  from  the  source  selection  being  surveyed.  The 
scale  was  adapted  from  a  study  conducted  on  buyer-seller  relationships  in  commercial  markets 
(Cannon  &  Perreault,  1999).  The  variable  consisted  of  five  questions  using  a  seven-point  Likert- 
type  scale  with  anchors  of  strongly  agree  and  strongly  disagree.  One  variable  from  this  construct 
was  discarded  as  a  result  of  factor  analysis. 
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Table  10.  Construct  of  Buyer  Satisfaction 


Scale  Item  Survey  Question 

BS^#*  Our  customer  regrets  the  decision  to  do  business  with  this  contractor. 

BS  Overall,  our  customer  is  very  satisfied  with  this  contractor. 

-1 

BS  Our  customer  is  very  pleased  with  what  the  contractor  does  for  the  customer. 

BS"^*  Our  customer  is  not  completely  happy  with  this  contractor. 

BS^#  If  we  had  to  do  it  all  over  again,  we  would  still  choose  to  use  this  contractor. 

#  item  was  discarded  due  to  Exploratory  Factor  Analysis 

*  item  was  reverse  coded 

Contractor  performance  is  the  overall  measure  of  the  successful  offeror’s  adherence  to 
contract  obligations.  Vetting  from  industry  practitioners  validated  the  contractor  performance 
construct.  Table  11  shows  the  seven  aspects  of  contract  execution  that  were  used  to  measure  the 
level  of  contractor  performance  that  resulted  from  the  contracts  awarded  from  the  source 
selections  for  which  respondents  answered  the  survey  questions.  The  scale  was  adapted  from 
past  marketing  and  operations  management  studies  (Fawcett,  Smith,  &  Cooper,  1997;  Cannon, 
Achrol,  &  Gundlach,  2000;  Prahinski  &  Benton,  2004).  The  variable  consisted  of  five  questions 
using  a  seven-point  Likert-type  scale  with  anchors  of  needs  improvement  and  superior 
performance. 
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Table  1 1 .  Construct  Measurement  of  Contractor  Performance 


Scale  Item 

Survey  Question 

“c? 

CP^ 

Product/service  quality  per  specifications 

Delivery  performance  per  specifications 

CP^ 

Product/service  consistently  meets  customer  expectations 

CP^ 

Responsiveness  to  requests  for  changes 

CP^ 

Required  service  and/or  technical  support 

CP^ 

Non-conformance  rate 

CP^ 

Overall  performance 

Vetting  from  industry  practitioners  validated  the  protest  risk  construct.  Protest  risk  was 
an  exploratory  construct  that  consists  of  two  parts  since  risk  is  comprised  of  the  product  of  the 
magnitude  of  the  consequences  and  the  probability  of  occurrence.  Table  12  shows  the  questions 
used  to  assess  the  desirability  of  each  of  five  consequences  and  each  of  their  associated 
probabilities  of  occurring.  The  five  consequences  were  validated  by  interview  informants  to  be 
those  most  likely  to  occur  and  those  most  abhorred.  Risk  was  calculated  by  multiplying  the 
consequences’  probability  of  occurrence  by  the  desirability  of  the  consequence.  The  result 
yielded  an  overall  protest  risk  score.  The  construct  consisted  of  five  possible  consequences  using 
a  Likert-type  scale  with  anchors  of  completely  undesirable  and  completely  desirable,  and  then 
listed  the  same  five  consequences  using  a  probability  of  occurance  scale  with  anchors  of  0 
percent  to  100  percent. 
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Table  12.  Construct  Measurement  of  Protest  Risk 


Scale  Item 

Survey  Question 

~P^ 

PR^ 

Increased  costs  to  settle  a  terminated  contract(s). 

Time  delay  to  the  mission. 

PR^ 

Embarrassment/shame. 

PR^ 

Increase  in  workload  to  resolve  the  protest. 

PR^ 

Career  repercussions  for  making  a  mistake  or  omission  that  caused  a  bid 
protest. 

risk  =  probability  of  occurrence  x  magnitude  of  consequences 


Source  selection  method  appropriateness  is  the  perceived  extent  that  the  chosen  source 
selection  method  fits  the  requirement,  the  goals  of  the  source  selection,  the  commercial  market, 
and  the  acquisition  situation.  The  source  selection  method  appropriateness  factor  is  new.  The 
factor  was  validated  by  qualitative  interviews.  The  decision  about  the  source  selection  method  to 
be  used  is  made  very  early  on  in  the  acquisition.  The  variable  consisted  of  six  questions  using  a 
seven-point  Likert-type  scale  with  anchors  of  strongly  disagree  and  strongly  agree,  shown  in 
Table  13. 
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Table  13.  Source  Selection  Method  Appropriateness 


Scale  Item _ Survey  Question _ 

SSMA^  Our  acquisition  strategy  was  the  best  means  to  source  our  requirement. 

SSMA  Our  acquisition  strategy  was  the  best  means  to  achieve  our  acquisition 

objectives. 

SSMA  It  would  have  been  difficult  to  achieve  our  goals  without  the  use  of  our 

acquisition  strategy. 

SSMA'^  The  source  selection  method  we  used  (e.g..,  LPTA,  full-trade-off,  or  PPT) 

was  the  most  appropriate  for  this  requirement. 

SSMA^  Our  acquisition  strategy  ensured  we  selected  the  best  offeror. 

SSMA^  Our  acquisition  strategy  provided  the  best  fit  to  the  buying  situation  (e.g., 

complexity,  dollar  value,  acquisition  objectives,  contract  length,  performance 
risk,  criticality  to  the  mission,  availability  of  supply,  time  available  to  award 

_ a  contract,  etc.). _ 


The  source  selection  method  fit  variable  is  an  exploratory  concept  that  had  not  been 
previously  tested.  Answers  to  the  questions  in  Table  14  were  used  to  explore  various  aspects  of 
the  overall  source  selection.  This  is  a  more  objective  measure  of  the  same  construct  in  Table  13, 
source  selection  method  appropriateness.  Source  selection  method  fit  measures  the  difference  in 
time  and  resources  used,  method  used,  and  satisfaction  with  discussions.  Measures  for  each 
question  are  listed  at  the  bottom  of  Table  14. 


Scale  Item 
SSMF^* 

SSMF^* 

SSMF^* 

SSMF^* 

SSMF^** 


Table  14.  Source  Selection  Method  Fit 

Survey  Question _ 

During  acquisition  planning,  how  many  days  were  planned  in  the 
milestones  from  receipt  of  a  complete  requirements  package  to  award 
of  the  contract(s)? 

Number  of  actual  days  from  receipt  of  a  complete  requirements 
package  to  award  of  the  contract: 

Number  of  people  on  the  source  selection  team  including  all  advisors, 
reviewers,  full-time  participants,  and  part-time  participants: 

Absent  a  risk  of  a  bid  protest,  in  your  opinion,  how  many  people  ideally 
should  have  been  on  the  source  selection  team: 

What  source  selection  method  was  used?  (Select  one.) 
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Scale  Item 

Survey  Question 

SSMF^*** 

Rate  the  appropriateness  of  each  source  selection  method  for  the 
requirement  on  a  scale  of  1  to  7,  where  1  represents  “completely 
inappropriate”  and  7  represents  “completely  appropriate.”  LPTA 
(Lowest  Price  Technically  Acceptable) 

ssmf’*** 

PPT  (Price-Past  Performance  Trade-off ) 

SSMF^*** 

Full  Trade-off 

ssmf‘^**** 

Rate  your  level  of  satisfaction  with  the  freedom  to  openly  discuss  those 
aspects  of  the  proposals  that  needed  to  be  discussed  with  the  offeror  in 
order  to  fully  understand  the  offer  and  to  properly  evaluate  the 

SSMF^°***** 

proposals. 

Hypothetically,  if  there  was  no  ability  to  protest,  rate  the  extent  that 
discussions  with  offerors  would  have  differed. 

ssmf"****** 

Did  the  RFP  advise  offerors  that  the  government  intended  to  award 
without  discussions? 

SSMF^^****** 

Did  you  conduct  discussions? 

SSMF^^*** 

Considering  the  risk,  criticality,  dollar  value,  contribution  to  the 
mission,  and  complexity,  rate  the  appropriateness  of  awarding  a 

SSMF^‘******* 

contract  without  conducting  discussions. 

Did  offerors  make  oral  presentations? 

SSMF^^*** 

Considering  the  risk,  criticality,  dollar  value,  contribution  to  the 
mission,  and  complexity,  rate  the  appropriateness  of  not  utilizing  oral 
presentations. 

*  Measured  by  text  input  into  a  field 
**  Used  a  drop  box  with  appropriate  choices 

***  Likert- type  scale  with  anchors  of  completely  appropriate  and 
completely  inappropriate 

****  Likert-type  scale  with  anchors  of  completely  satisfied  and 
completely  dissatisfied 

*****  Likert-type  scale  with  anchors  of  no  difference  and  substantial 
difference 

******  Choices  of  Yes  or  No 

The  number  of  awarded  contracts  variable  was  validated  by  qualitative  interviews.  The 
measure  was  used  to  analyze  the  difference  between  the  number  of  contract  awards  that  were 
planned  and  the  number  of  contracts  actually  awarded.  These  were  objective  measures  along  a 
continuous  scale.  The  questions  used  are  shown  in  Table  15. 
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Table  15.  Measurement  of  Number  of  Awarded  Contracts 


Scale  Item _ Survey  Question _ 

AC’  According  to  your  acquisition  planning,  how  many  contracts  were  originally 

intended  to  be  awarded? 

2 

AC _ How  many  contracts  were  actually  awarded? _ 


The  transaction  costs  variable  is  an  objective  measure  that  was  used.  This  measure 
consisted  of  25  items,  shown  in  Table  16  and  was  validated  through  qualitative  interviews  and 
discussions  with  other  industry  practitioners.  The  first  10  items  gathered  the  number  of  iterations 
of  common  source  selection  documents.  They  were  represented  by  the  abbreviation  TCI.  The  last 
15  attempted  to  quantify  in  dollars  the  amount  of  time  spent  on  the  source  selection  by  each 
member  of  the  team,  and  were  represented  by  the  abbreviation  TCP.  The  roles  were  assigned 
using  information  from  the  DOD  source  selection  procedures  (OUSD  [AT&L],  2011).  There  was 
no  previously  validated  scale  to  measure  these  transaction  costs.  Text  boxes  to  gather  data  were 
used  to  answer  these  questions. 

Table  16.  Construct  of  Transaction  Costs 

Scale  Item _ Survey  Question _ 

TCI^  During  the  source  selection,  how  many  iterations  of  each  of  the  following 

documents  were  generated? 

Source  selection  decision  document 
TCI^  Comparative  assessment/proposal  analysis  report 

TCI  Evaluation  notices  (sum  of  iterations  for  all  ENs  across  all  offerors) 

TCl"^  Source  selection  plan 

TCI^  Debriefing  scripts  (sum  of  iterations  for  all  scripts  across  all  offerors) 

TCI^  Technical  evaluations  (sum  of  iterations  for  all  evaluations  across  all 

offerors) 

n 

TCI  Past  performance  evaluations  (sum  of  iterations  for  all  evaluations  across  all 

offerors) 

TCI^  Cost/price  analysis 
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Scale  Item _ Survey  Question _ 

TCI^  Color/rating  chart 

TCI*°  Evaluation  briefing  charts  for  reviewers  and  SS 

TCP^  For  the  following  roles,  please  provide  the  number  of  people  within  that  role, 

their  grade(s)/rank(s),  and  the  fraction  of  1  year’s  time.  If  no  involvement, 
just  skip  to  the  next  role.  Contracting  Officer 
TCP^  Contract  specialists 

TCP^  Technical  Evaluators 

TCP"*  Cost/price  analysts 

TCP^  Supervisors 

TCP^  Eegal 

TCP’  Past  performance  team 

o 

TCP  Source  selection  advisory  council/non-legal  advisors 

TCP^  Source  selection  authority 

TCP'°  Financial  management 

TCP'^  Program  manager/requirements  office 

TCP  Small  business  representative 

TCP  Contracted  consultants/labor 

TCP*"^  Other  (description) 

TCP^^ _ Other  (data  field) _ 

The  competence  factor  was  validated  by  qualitative  interviews  and  discussions  with 
industry  practitioners  as  shown  in  Table  17. 
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Table  17.  Construct  Measurement  of  Contracting  Officer  Competence 


Scale  Item _ Survey  Question _ 

”C^  How  many  total  years  of  experience  do  you  have  in  contracting? 

C^  In  how  many  source  selections  have  you  participated  throughout  your  career? 

E.  SURVEY  PRE-TEST 

Six  industry  practitioners,  which  consisted  of  graduate-level  students  and  professors  who 
specialize  in  DOD  contracting  tested  the  initial  survey.  Feedback  received  was  used  to  refine 
questions  and  limit  survey  length.  Responses  supported  the  proposed  research  hypothesis,  but  in 
order  to  shorten  the  survey,  one  antecedent — risk  tolerance — was  removed. 

Once  constructed,  the  order  of  the  survey  questions  was  structured  to  reduce  bias  among 
scale  items  by  mixing  questions  with  like  scales  and  scale  anchors.  The  complete  survey  can  be 
found  in  Appendix  A.  As  required,  the  survey  was  reviewed  and  approved  by  the  Navy  Survey 
Office.  An  additional  review  was  conducted  through  the  Department  of  the  Navy’s  Institutional 
Review  Board  to  ensure  the  protection  of  human  subjects. 

F.  RESEARCH  DESIGN 

This  study  attempts  to  answer  the  following  questions  via  qualitative  and  quantitative 
methods: 

•  Does  a  fear  of  protest  exist  in  the  acquisition  workforce? 

•  If  so,  what  are  the  contributing  factors? 

•  If  so,  what  are  the  outcomes? 

A  conceptual  model  that  included  16  variables  was  developed  by  the  research  team  to 
identify  those  factors  that  contributed  to  protest  fear  and  hypothesize  what  verifiable  results 
could  be  observed.  The  model  was  then  vetted  by  discussion  with  22  current  and  previously- 
warranted  contracting  officers  and  three  Air  Force  and  Navy  contracting  unit  leaders  for  content, 
or  face,  validity. 

To  precisely  identify  and  measure  the  relationships  shown  in  the  conceptual  model, 
quantitative  analysis  through  hypothesis  testing  between  each  contributing  and  resulting  variable 

and  fear  of  protest  was  performed.  The  path  diagram  of  the  conceptual  model  (see  Figure  1) 
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contains  14  relationships  that  represent  the  14  hypotheses.  The  relationships  between  variables 
are  depicted  using  arrows;  “the  directionality  of  causal  paths  begins  from  a  variable  at  the 
arrow’s  tail  and  ends  at  a  variable  about  the  arrow’s  point”  (Muir,  2010,  p.  38).  These  arrows 
contain  a  plus  or  minus  sign  to  reflect  the  expected  direction  of  the  relationship.  The  primary 
methods  used  to  perform  statistical  analysis  are  simple  and  multiple  linear  regressions. 

Linear  regression  tries  to  model  the  relationship  between  two  variables  by  fitting  a  linear 
equation  to  data  on  the  X-  and  Y-axis.  “The  case  of  one  dependent  and  independent  variable  is 
called  simple  linear  regression.  For  more  than  one  explanatory  variable,  it  is  called  multiple 
linear  regression”  (“Linear  Regression,”  1998). 

In  linear  regression,  data  is  modeled  “using  linear  predictor  functions,  and  unknown 
model  parameters  are  estimated  from  the  data”  (Stanton,  2001,  p.  237).  The  result  is  known  as  a 
linear  model.  Most  commonly,  “linear  regression  refers  to  a  model  in  which  the  conditional 
mean  of  Y  given  the  value  of  X  is  an  affine  function  of  X”  (Cohen,  West,  &  Aiken,  2003,  p.  42). 
The  purpose  of  causal  modeling  is  to  predict  or  explain  phenomenon.  Causal  models  are 
“appropriate  for  domains  in  which  many  modeled  events  can  be  conceptualized  as  processes 
causing  other  events  which  in  turn  trigger  other  processes”  (Lemmer,  1993,  p.  143). 

Multiple  regression  is  a  form  of  causal  modeling  that  can  be  used  to  analyze  the 
relationship  between  one  dependent  variable  and  several  independent  variables.  The  “flexibility 
and  adaptability  of  multiple  regression  allow  its  use  with  almost  any  dependence  relationship” 
(Hair,  Black,  Babin,  &  Anderson  p.  166).  Multiple  regression  can  be  made  applicable  to  a  wide 
range  of  problems  because  of  this  flexibility.  Its  application  is  appropriate  for  two  types  of 
research  problems:  prediction  and  explanation.  Prediction  deals  with  the  magnitude  that  the 
independent  variable  can  predict  the  value  of  the  dependent  variable.  Explanation  uses  regression 
coefficients  for  the  independent  variables  and  “attempts  to  develop  a  substantive  or  theoretical 
reason  for  the  effects  of  the  independent  variables”  (Hair  et  ah,  p.  167). 

The  goal  of  this  research  is  not  to  predict  the  level  of  the  fear  dependent  variable,  but  to 
explain  the  significance  of  its  relationships  with  the  independent  variables  that  are  used.  Multiple 
regression  is  the  appropriate  tool  to  answer  the  questions  being  asked  because  it  can  objectively 
asses  the  magnitude  and  character  of  the  relationships  of  the  conceptual  model.  When  regression 
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analysis  is  performed,  it  is  assumed  that  there  will  be  a  linear  association  between  the  dependent 
and  independent  variables.  With  multiple  regression,  “transformations  or  additional  variables  are 
available  to  assess  whether  other  types  of  relationships  exist,  particularly  curvilinear 
relationships.  This  flexibility  ensures  that  the  researcher  may  examine  the  true  nature  of  the 
relationship  beyond  the  assumed  linear  relationship”  (Hair  et  al.,  p.  168). 

G.  CONCLUSION 

In  this  chapter,  the  qualitative  interviews  that  occurred  were  discussed.  Next,  the 
population  and  sampling  methods,  followed  by  the  survey  design  and  construct  measurement 
was  discussed.  Finally,  the  research  design  and  statistical  method  used  for  the  data  analysis  was 
discussed.  In  the  next  chapter,  the  results  of  the  data  collection  and  statistical  analysis  are 
presented. 
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IV.  RESULTS 


A.  INTRODUCTION 

This  chapter  presents  the  results  of  the  survey,  hypothesis  testing,  and  conceptual  model. 
The  data  collection  results  are  discussed  first,  followed  by  the  sample  demographics.  The  results 
of  tests  for  normality,  outliers,  non-response  bias,  reliability,  and  validity  are  then  discussed. 
Next,  the  results  of  the  exploratory  factor  analysis  (EFA)  are  performed  in  SPSS  Statistics 
Version  22.0.  The  results  of  the  regression  analyses  performed  are  then  discussed.  Finally,  the 
conceptual  model  with  all  hypothesized  relations  is  presented. 

When  analyzing  the  significance  of  the  relationship  between  constructs,  the  path 
coefficient  is  used  to  measure  strength.  Alpha  levels  that  are  less  than  .05  are  considered  to  have 
a  strong  relationship  between  dependent  and  independent  variables. 

B.  DATA  COLLECTION 

Data  was  collected  from  a  NPS-approved  web-based  survey  tool.  Lime  survey.  A  survey 
invitation  (Appendix  B)  was  sent  via  e-mail  to  contracting  officers  who  had  executed  a  FAR  Part 
15-based  formal  source  selections  with  a  dollar  value  greater  than  or  equal  to  $150,000  during 
FY2013.  E-mail  addresses  were  extracted  from  the  FPDS-NG  database.  A  memorandum  of 
support  from  the  Deputy,  Assistant  Secretary  of  the  Navy  (Acquisitions  and  Procurement)  was 
referred  to  in  the  e-mail  to  notify  the  respondents  of  sponsorship  (Appendix  C).  The  survey  was 
left  active  for  42  days,  with  three  reminder  invitations  being  sent  during  that  time.  The  survey 
was  sent  to  a  population  of  4,279  unique  e-mail  addresses.  Of  these  unique  addresses,  260 
undeliverable  responses  were  received,  and  137  individuals  replied  stating  that  they  had  not 
participated  in  a  FAR  Part  15  Source  Selection.  As  a  result,  the  total  population  was  reduced  to 
3,882.  At  the  survey  closure,  there  were  661  responses  received,  which  yielded  a  17.02  percent 
response  rate.  Of  the  responses  received,  311  had  to  be  deleted  because  of  missing  or  invalid 
data,  leaving  350  usable  responses.  The  final  response  rate  of  usable  responses  is  9.01  percent. 

While  the  response  rate  is  low,  it  is  not  unlike  that  of  other  published  business  research. 

In  a  study  focused  on  declining  response  rates  in  supply  chain  management  research,  Melnyk, 

Page,  Wu,  and  Bums  (2012)  stated  that  “the  issue  of  declining  response  rates  is  a  growing 
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concern”  (p.  35).  Their  research  showed  a  sharp  decline  starting  in  2002,  with  a  steady  decline  of 
one  percent  annually.  Five  top  journals  reported  low  end  survey  response  rates  ranging  from 
three  percent  to  eight  percent.  Survey  length  is  thought  to  be  a  one  of  the  key  contributors  to  the 
decline.  Melnyk  et  al.  further  stated  that  “for  every  additional  question  over  20,  the  researcher 
can  expect  the  response  rate  to  fall  by  0.12  percentage  points.  Consequently,  the  researcher  can 
expect  the  response  to  fall  by  over  seven  percentage  points  in  response  to  an  80  question 
survey.”  (Melnyk  et  al.,  2012,  p.  43)  The  summary  descriptive  statistics  from  the  remaining  350 
usable  responses  can  be  found  in  Table  18. 
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Table  18.  Summary  Statistics 


Variable  Measure 

N  Minimum  Maximum 

Mean 

Std.  Deviation 

Variable 

Measure  N 

Minimum 

Maximum 

Mean 

Std.  Deviation 

Fear  FEARl 

3S0 

1 

7 

4.37 

1.919 

Protest  Risk 

PRl  3S0 

1 

11 

3.44 

2.86 

FEAR2 

3S0 

1 

7 

4.22 

1.887 

PR2  3S0 

1 

11 

2.S3 

2.294 

FEAR3 

3S0 

1 

7 

4.41 

1.9S9 

PR3  3S0 

1 

11 

S.42 

3.114 

FEAR4 

3S0 

1 

7 

3.82 

1.76S 

PR4  3S0 

1 

10 

3.S1 

2.87 

FEARS 

3S0 

1 

7 

3.33 

1.666 

PRS  3S0 

1 

11 

4.63 

3.21S 

FEAR6 

3S0 

1 

7 

3.38 

1.708 

PRl  3S0 

0.00% 

100.00% 

48.17% 

31.01% 

PR2  3S0 

0% 

100% 

7S.09% 

28.2S% 

Contracting  Officer  Authority  CO_AUTH  1 

3S0 

1 

7 

S.27 

1.S46 

PR3  3S0 

0.00% 

100.00% 

36.09% 

31.S7% 

CO_AUTH2 

3S0 

1 

7 

S.43 

1.4S2 

PR4  3S0 

0% 

100% 

74.26% 

28.SS% 

CO_AUTH3 

3S0 

1 

7 

S.6 

1.341 

PRS  3S0 

0.00% 

100.00% 

33.23% 

30.27% 

CO_AUTH4 

3S0 

1 

7 

3.84 

1.703 

Source  Selection 

CO_AUTH5 

3S0 

1 

7 

4.17 

1.887 

Method  Appropriateness  SSMAl  350 

1 

7 

S.Sl 

1.31S 

CO_AUTH6 

3S0 

1 

7 

3.1 

1.7S9 

SSMA2  3S0 

1 

7 

S.61 

1.243 

SSMA3  3S0 

1 

7 

S.17 

1.S32 

PALT  Planned  PALT_P1 

3S0 

1 

7 

4.38 

1.79S 

SSMA4  3S0 

1 

7 

S.76 

1.292 

PALT_P2 

3S0 

1 

7 

3.36 

1.764 

SSMAS  3S0 

1 

7 

S.S2 

1.339 

PALT_P3 

3S0 

1 

7 

3.83 

1.7S 

SSMA6  3S0 

1 

7 

S.Sl 

1.39S 

SSMA7  3S0 

0 

1000000 

3168.97 

S3702.779 

Technical  Evaluation  Effectiveness  TEEl 

3S0 

1 

7 

S.33 

1.S84 

SSMAS  3S0 

0 

200 

2.89 

14.2S2 

TEE2 

3S0 

1 

7 

3.S 

1.714 

SSMA9  3S0 

0 

ISO 

3.04 

13.S74 

TEE3 

3S0 

1 

7 

3.37 

1.628 

Source  Selection 

TEE4 

3S0 

1 

7 

4.37 

1.434 

Method  Fit 

SSMEl  3S0 

0 

780 

182.97 

131.168 

TEES 

3S0 

1 

7 

3.03 

1.817 

SSMP2  3S0 

0 

3290 

236.72 

244.081 

TEE6 

3S0 

1 

7 

3.47 

1.744 

SSMP3  3S0 

0 

138 

8.96 

10.0S2 

SSMP4  3S0 

0 

130 

7.6S 

9.0S3 

Requiement  Criticality/Importance  RCIl 

3S0 

1 

7 

S.9 

1.093 

SSMES  3S0 

1 

3 

2.12 

0.931 

RCI2 

3S0 

1 

7 

S.79 

1.13S 

SSMP6  3S0 

1 

7 

4.16 

2.181 

RCI3 

3S0 

1 

7 

5.11 

1.142 

SSMP7  3S0 

1 

7 

3.82 

1.776 

RCI4 

3S0 

1 

7 

2.41 

1.423 

SSMP8  3S0 

1 

7 

4.89 

2.061 

RCIS 

3S0 

1 

7 

4.1S 

1.666 

SSMP9  3S0 

1 

7 

3.28 

1.718 

RCI6 

3S0 

1 

7 

4.1S 

1.668 

SSMFIO  3S0 

1 

7 

3.37 

2.041 

SSMF13  3S0 

1 

7 

4.2 

2.113 

Buyer  Satisfaction  BSl 

3S0 

1 

7 

2.44 

1.3S9 

SSMFIS  3S0 

1 

7 

4.7 

2.099 

BS2 

3S0 

1 

7 

S.S 

1.232 

Actual  Number  of 

BS3 

3S0 

1 

7 

S.46 

1.176 

Contracts  Awarded 

ACl  3S0 

0 

60 

2.24 

4.879 

BS4 

3S0 

1 

7 

3.02 

1.474 

AC2  3S0 

0 

ss 

2.39 

S.S06 

BSS 

3S0 

1 

7 

S.61 

1.22S 

Contractor  Performance  CPI 

3S0 

1 

7 

4.69 

1.372 

CP2 

3S0 

1 

7 

4.74 

1.32 

CP3 

3S0 

1 

7 

4.74 

1.32 

CP4 

3S0 

1 

7 

4.76 

1.303 

CPS 

3S0 

1 

7 

4.7S 

1.261 

CP6 

3S0 

1 

7 

4.S3 

1.213 

CP7 

3S0 

1 

7 

4.88 

1.223 
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C.  SAMPLE  DEMOGRAPHICS 

From  the  350  usable  responses,  the  average  respondent  had  13.6  years  of  federal 
contracting  experience.  There  were  314  respondents  who  elected  to  identify  their  gender.  Results 
were  even  with  male  respondents  accounting  for  50.64  percent  and  female  respondents 
accounted  for  49.36  percent.  The  majority  of  respondents  represented  had  at  least  a  bachelor’s 
degree.  Only  2.29  percent  of  respondents  held  a  high  school  diploma  or  general  equivalency 
diploma,  3.14  percent  held  an  associate’s  degree,  38  percent  held  a  bachelor’s  degree,  54  percent 
held  a  master’s  degree,  and  only  2.57  percent  of  respondents  held  a  doctorate  degree.  As  to  be 
expected,  respondents’  Acquisition  Professional  Development  Program  (APDP)  certification 
levels  in  contracting  were  skewed  to  higher  levels.  Typically,  more  experienced  contracting 
professionals  conduct  FAR  Part  15  source  selections.  Only  1.43  percent  of  the  respondents  held 
no  APDP  contracting  certification,  2.86  percent  held  a  Level  I  APDP  certification,  37.43  percent 
held  a  Level  II  APDP  certification,  and  58.29  percent  held  a  Level  III  APDP  certification. 
Demographics  for  the  education  and  certification  levels  of  the  respondents  can  be  found  in  Table 
19. 


Table  19.  Education  and  Certification  Demographics 


Education  Level 

%  of  Total 

Certification  Level 

%  of  Total 

High  School  Diploma  of  GED 

2.29% 

No  Certification 

1.43% 

Associate’s  Degree 

3.14% 

Level  I 

2.86% 

Bachelor’s  Degree 

38.00% 

Level  II 

37.43% 

Master’s  Degree 

54.00% 

Level  III 

58.29% 

Doctoral  or  professional  degree 

2.57% 

Respondents  were  asked  to  refer  back  to  the  most-recent  completed  source  selection 
above  $150,000  in  which  they  had  participated.  The  range  of  contract  type  and  type  of 
products/services  are  reported  (see  Table  21).  Of  the  respondents,  60  percent  reported  the  use  of 
fixed-priced,  27  percent  reported  cost  reimbursement,  one  percent  reported  Time  and  Materials, 
one  percent  reported  labor-hour,  nine  percent  reported  hybrid,  and  two  percent  reported  as  other. 
The  type  of  supply  or  service  bought  was  also  reported.  Services  were  bought  in  58  percent  of 
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the  cases,  construction  made  up  17  percent  of  the  purchases,  supplies  or  commodities  made  up 
16  percent,  weapons  systems  made  up  seven  percent,  and  other  capital  equipment  made  up  three 
percent.  Experience-related  demographics  information  can  be  found  in  Table  20. 


Table  20. 

Experience 

Years  of  Experience 

Total 

Other  Statistics 

Total 

Average  of  all  respondents 

13.12 

Avg  Number  of  Source  Selections 

18.31 

Respondents  with  5  or  less  years 

110(31.4%) 

Avg  Number  of  Protests  experienced 

1.96 

Respondents  with  between  5  and  15 
years 

Respondents  with  greater  than  15 
years 

130  (37.1%) 

120  (34.3%) 

Number  (%)  who  had  a  protest 
sustained 

54  (15.4%) 

Table  21.  Contract  Type  and  Item/Service  Procured 


Contract  Type 

%  of  Total 

Supply  or  Service  Purchased 

%  of  Total 

Fixed  Price 

60.00% 

Services 

58.00% 

Cost  Reimbursement 

27.14% 

Construction 

16.86% 

Time  and  Materials 

1.14% 

Supplies  or  Commodities 

15.71% 

Labor  Hours 

0.57% 

Weapons  System 

6.57% 

Hybrid 

9.14% 

Other  Capital  Equipment 

2.86% 

Other 

2.00% 

D.  ASSUMPTIONS 

1.  MISSING  DATA 

Out  of  661,  there  were  311  responses  discarded  because  of  missing  data,  leaving  a  total 
of  350  usable  responses.  Of  these  responses,  36  did  not  include  a  properly  reported  gender 
answer;  these  respondents  were  not  included  in  the  gender  demographics  reporting.  The 
Transaction  Cost  variable  had  multiple  missing  and  inconsistent  items  of  data  since  this  was  an 
onerous  task  for  respondents  to  complete,  and  since  it  was  an  optional  part  of  the  survey. 
Additionally,  the  survey  collected  the  pay  grade  and  allocation  of  time  via  open  text  fields 
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enabling  inconsistent  response  formats.  All  reporting  of  this  data  includes  only  the  complete 
responses  with  unambiguous  data.  Only  270  responses  were  able  to  be  used  for  this  analysis. 

2.  OUTLIERS 

Outliers  are  classified  as  “a  unique  combination  of  characteristics  identifiable  as 
distinctly  different  from  other  observations”  (Hair  et  al.,  p.  63).  Outliers  such  as  procedural  error, 
extraordinary  events,  and  extraordinary  observations  can  arise  for  a  variety  of  reasons.  .  Most  of 
the  questions  contained  in  this  survey  were  on  a  Likert-type  scale  where  the  ranges  of  responses 
were  limited. 

The  presence  of  Univariate  and  Multivariate  outliers  was  tested  using  SPSS  version  22. 
To  conduct  the  Univariate  test,  the  score  of  10  variables  were  converted  to  standardized  values, 
or  z-scores.  Then,  the  standard  value  was  evaluated  against  a  benchmark  in  which  potential 
outliers  had  the  z-score  of  +3.00  or  higher  (Muir,  2010,  p.  56).  The  range  of  standard  scores  can 
be  seen  in  Table  22.  Five  variables  had  cases  that  were  above  or  below  the  threshold.  Contracting 
officer  authority  had  five  cases  that  were  outside  of  the  threshold;  Requirement  Criticality  or 
Importance  had  three  cases  that  were  outside  of  the  threshold;  Buyer  Satisfaction  had  four  cases 
that  were  outside  of  the  threshold;  Contractor  Performance  had  three  cases  that  were  outside  of 
the  threshold;  and  Source  Selection  Method  Appropriateness  had  seven  cases  that  were  outside 
of  the  threshold.  Each  case  was  examined  for  inconsistent  responses.  This  examination  did  not 
lead  to  any  responses  being  discarded. 

Since  multivariate  detection  involves  several  variables,  the  ability  to  measure  the 
multidimensional  position  of  each  response  relative  to  a  common  point  is  needed.  The 
Mahalanobis  Distance,  or  D  ,  is  a  multivariate  assessment  of  each  observation  across  a  set  of 
variables  (Hair  et  ah,  2010).  The  same  10  variables  were  used  to  calculate  the  D  in  SPSS.  The 
critical  value  was  29.59  when  10  variables  were  used.  Five  cases  exhibited  a  D  score  higher  than 
29.59.  Each  case  was  examined  for  inconsistencies  or  incoherent  responses.  This  examination 
also  led  to  no  responses  being  discarded.  The  Transaction  Cost  document  iteration  variable  was 
also  examined  for  incoherent  responses.  Twelve  outlying  responses  were  truncated  to  the  highest 
reasonable  score.  “One  alternative  to  transformation  is  truncation,  wherein  extreme  scores  are 
recorded  to  the  lowest  reasonable  score”  (Osborne  &  Overbay,  2004,  p.  2).  Reasonable  scores 
were  determined  using  the  average  number  of  iterations  for  each  document.  If,  for  example,  a 
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response  indicated  that  the  respondents  experienced  99  iterations  of  a  document,  this  was 
considered  an  unrealistic  response.  This  response  was  then  truncated  to  the  average  iteration 
number  for  the  subject  document. 


Table  22.  Univariate  Range  of  Standard  Scores 


Variable 

N 

Minimum 

Maximum 

Zscore(FEAR) 

350 

-1.88982 

1.96546 

Zscore(CO_AUTH)  350 

-3.45073 

1.13423 

Zscore(PALT_P) 

350 

-1.73510 

2.04099 

Zscore(TEE) 

350 

-1.75669 

2.55294 

Zscore(RCI) 

350 

-4.63729 

1.18532 

Zscore(BS) 

350 

-4.72418 

4.14633 

Zscore(CP) 

350 

-3.19421 

1.94928 

Zscore(SSMA) 

350 

-4.06658 

1.34293 

3.  NORMALITY 

Normality  is  the  most  fundamental  assumption  in  multivariate  analysis  and  the 
benchmark  for  statistical  methods.  If  the  variation  from  the  normal  distribution  is  sufficiently 
large,  all  resulting  statistical  tests  are  invalid  (Hair  et  ah,  2010).  The  shape  of  normally 
distributed  data  is  a  symmetric  bell-shaped  curve.  A  distribution  can  be  described  by  two  basic 
measures,  skewness  and  kurtosis.  Skewness  describes  the  lateral  balance  of  the  distribution.  A 
positive  skew  describes  a  distribution  that  is  shifted  left,  while  a  negative  skew  is  shifted  to  the 
right.  Kurtosis  denotes  the  peak  or  flatness  of  a  distribution  when  compared  to  a  normal  bell 
shaped  distribution.  Distributions  with  a  higher,  or  more  peaked,  distribution  are  said  to 
leptokurtic,  while  flatter  distributions  are  said  to  be  platykurtic  (Hair  et  ah,  2010). 

Graphical  analysis  of  a  distribution  using  a  histogram  is  the  easiest  test  for  normality. 
However,  this  method  can  be  less  reliable  with  small  sample  sizes.  The  normal  probability  plot  is 
the  most  reliable  way  to  do  a  visual  assessment  of  normality.  The  normal  probability  plot 
compares  the  distribution  of  the  actual  data  with  distribution  of  a  normal  distribution.  The 
normal  distribution  line  is  diagonal  and  the  plotted  data  is  compared  to  the  normal  distribution 
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line.  The  Shapiro- Wilks  test  is  another  common  normality  test.  This  test  is  very  powerful  for 
smaller  sample  sizes  (~50)  but  tends  to  be  overly  sensitive  for  samples  of  over  200  (Totton  & 
White,  2011). 

Table  22  displays  the  statistics  for  skewness  produced  using  descriptive  statistics  function 
in  SPSS.  Values  above  and  below  zero  denote  non-normality  (Hair  et  ah,  2010).  While  skewness 
scores  of  zero  represent  a  perfectly  normal  distribution,  moderately  non-normal  distributions  are 
represented  by  scores  greater  than  2.0  (Curran,  West,  &  Finch,  1995).  Using  a  range  for 
skewness  of  +2.0,  Table  22  showed  nine  variables  that  exceeded  this  threshold.  In  order  to 
ensure  the  validity  of  statistical  analysis,  none  of  the  variables  were  included  in  any  constructs 
that  were  used  to  perform  regression  analysis. 

Table  23  also  displays  statistics  for  kurtosis.  Normal  distributions  display  a  standardized 
kurtosis  value  of  3.0,  and  rescaled  distributions  display  a  value  of  zero  (Hair  et  ah,  2010). 
Curran,  West,  and  Finch  (1995)  suggested  that  rescaled  kurtosis  values  greater  than  or  equal  to 
seven  are  indicative  of  moderate  non-normality.  The  table  shows  nine  variables  that  exceeded 
this  threshold.  In  order  to  ensure  the  validity  of  statistical  analysis,  none  of  these  variables  were 
included  in  any  constructs  that  were  used  to  perform  regression  analysis. 
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Table  23.  Normality  Assessment 


Variable 

N 

Skewness 

Kurtosis 

Variable 

N 

Skewness 

Kurtosis 

Variable 

N 

Skewness 

Kurtosis 

FEARl 

350 

-.426 

-1.1 13 

CPI 

350 

-.503 

.401 

TCI3*** 

350 

7.940 

77.095 

FEAR2 

350 

-.331 

-1.155 

CP2 

350 

-.334 

.252 

TCI4 

350 

1.098 

4.757 

EEAR3 

350 

-.418 

-1.139 

CP3 

350 

-.409 

.145 

TCI5 

350 

1.409 

5.057 

EEAR4 

351 

.133 

-1.042 

CP4 

350 

-.175 

.054 

TCI6*** 

350 

2.853 

13.738 

EEAR5 

350 

.367 

-.597 

CP5 

350 

-.339 

.487 

TCI7*** 

350 

2.737 

14.525 

EEAR6 

350 

.369 

-.582 

CP6 

350 

-.014 

.572 

TCI8*** 

350 

2.758 

16.025 

CP7 

350 

-.266 

.095 

TCI9 

350 

.584 

.595 

CO_AUTHl 

350 

-1.089 

.571 

TCI  10 

350 

.050 

.032 

CO_AUTH2 

350 

-1.238 

1.108 

SSMAl 

350 

-1.205 

1.163 

CO_AUTH3 

350 

-1.390 

1.909 

SSMA2 

350 

-1.395 

2.221 

D_VAL 

350 

-.084 

-1.086 

CO_AUTH4 

350 

-.089 

-1.028 

SSMA3 

350 

-.916 

.039 

CO_AUTH5 

47 

-.096 

-1.077 

SSMA4 

350 

-1.649 

2.729 

CO_AUTH6 

350 

-.506 

-.723 

SSMA5 

350 

-1.201 

1.130 

SSMA6 

350 

-1.368 

1.529 

PALT_P1 

350 

.105 

-1.152 

SSMA7*** 

350 

18.447 

343.056 

PALT_P2 

350 

.303 

-1.056 

SSMA8**=*= 

350 

9.659 

115.466 

PALT_P3 

350 

-.050 

-1.201 

SSMA9*** 

350 

7.355 

62.431 

TEEl 

47 

-1.668 

2.098 

SSMF1=*= 

350 

-.042 

-.290 

TEE2 

350 

.357 

-1.074 

SSMF2* 

350 

1.027 

5.386 

TEE3 

350 

.370 

-.925 

SSMF3=*= 

350 

1.446 

4.539 

TEE4 

350 

.162 

-.532 

SSMF4* 

350 

1.726 

6.957 

TEE5 

350 

-.601 

-.830 

SSMF5 

50 

-.290 

-1.869 

TEE6 

350 

.322 

-1.129 

SSMF6 

50 

.100 

-1.555 

SSMF7 

50 

-.321 

-.552 

RCIl 

47 

-1.336 

2.326 

SSMF8 

50 

-.748 

-.466 

RCI2 

350 

-1.332 

2.195 

SSMF9 

50 

.378 

-.750 

RCI3 

350 

-1.181 

1.656 

SSMFIO 

50 

.181 

-1.423 

RCI4 

350 

1.198 

.730 

SSMF13 

50 

-.342 

-1.229 

SSMF15 

50 

-.682 

-.738 

BSl 

47 

.533 

-1.084 

Cl 

350 

.757 

-.696 

BS2 

350 

-1.270 

1.604 

C2** 

350 

1.434 

6.889 

BS3 

350 

-1.153 

1.502 

BS4 

350 

-.636 

-.395 

TCI* 

350 

.725 

5.283 

BS5 

47 

-.492 

-.990 

TCIl*** 

350 

4.720 

41.777 

TCI2*** 

350 

3.668 

38.750 

*  Transformed  using  the  square  root 

^^Transformed  using  the  LOG 

***Items  that  could  not  be  normalized,  and  were  not  included  in  any  tests 
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4. 


NON-RESPONSE  BIAS 


The  extrapolation  method  was  used  to  estimate  bias.  This  is  the  most  commonly  used 
method  when  responses  are  received  in  “successive  waves”  (Armstrong  &  Overton,  1977).  The 
responses  were  received  in  three  successive  waves  after  follow-up  e-mail  reminders  were  sent  to 
non-respondents.  Responses  were  grouped  into  waves  according  to  the  order  of  arrival.  Late 
respondents  are  defined  as 

those  who  respond  in  the  last  wave  of  respondents  in  successive  follow-ups  to  a 
questionnaire.  In  order  to  ensure  that  the  number  of  late  respondents  is  large 
enough  to  be  meaningful  practically  and  statistically,  it  is  recommended  that  the 
minimum  number  of  late  respondents  be  30.  (Linder,  2001  p.  52) 

If  the  tests  performed  on  this  method  does 

not  produce  sufficient  statistical  evidence  to  reject  the  null  hypotheses  that  the 
means  are  equal  then  it  is  not  unreasonable  to  assert  that  the  unobserved  group  of 
non-respondents  has  similar  characteristics  to  those  of  the  survey  respondents  and 
that  non-respondents  are  missing  at  random.  (Muir,  2010,  p.  60) 

Based  on  the  completion  date  of  the  survey,  respondents  were  grouped  into  three 
categories.  Group  one  (n  =  113)  represents  those  that  responded  within  the  first  two  days  of  the 
survey.  Group  two  (n  =  136)  represents  those  that  responded  after  the  second  follow  up  e-mail 
was  sent,  and  Group  three  (n  =  101)  represents  those  that  responded  after  the  final  follow  up  e- 
mail  was  sent.  Multiple  analysis  of  variance  (MANOVA)  was  then  conducted  on  nine  latent 
variables  to  test  the  difference  between  means  by  early,  middle,  and  late  responders.  Each 
individual  non-response  hypothesis  is  listed  in  Table  24. 
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Table  24.  Non-Response  Bias  Hypothesis 


Variable 

Hypothesis 

Fear 

Hq  ~  Ppearl  ~  PpearZ  ~  FFearB 

Contracting  Officer  Authority 

Hq  =  FcOAuthl  =  FC0Auth2  =  FCO  AuthB 

Planned  PALT 

Ho  =  Fpianned  PALTl  —  FPlanned  PALT2  =  FPlanned  PALT3 

Quality  of  Evaluation  Factors 

Hq  =  FtEEI  =  FtEE2  =  FTEE3 

Requirement 

Ho  =  FrCII  =  FrCI2  =  FRCI3 

Criticality /Importance 

Buyer  Satisfaction 

Ho  =  Fbsi  —  FbS2  —  FBS3 

Contractor  Performance 

Ho  =  FctrPerfl  —  Fctr  Perf2  —  Fctr  Perf3 

Source  Sel.  Method 

Ho  =  FsSMAI  =  FsSMA2  =  FSSMA3 

Appropriateness 

Protest  Risk 

Ho  ~  FProtest  Riskl  ~  FProtest  Risk2  ~  FProtest  Risk3 

Levene’s  test  was  calculated  for  each  variable.  The  results  are  listed  in  Table  25.  Since  no 
statistic  was  significant  at  a  =  0.05,  then  the  null  hypotheses  cannot  be  rejected.  Hence,  there  is 
little  evidence  that  the  variances  are  not  equal,  and  homogeneity  of  variances  can  be  assumed  for 
all  nine  variables.  Results  of  the  univariate  F-tests  are  shown  in  Table  26.  An  examination  of  the 
significance  of  the  F-test  statistics  reveals  that  none  are  significant  at  a  level  of  0.05.  As  a  result, 
there  is  no  statistical  evidence  to  reject  the  null  hypothesis  that  group  averages  are  equal  in  the 
nine  variables,  and  it  can  be  assumed  that  no  bias  exists  between  survey  responders  and  survey 
non-responders  in  these  cases  (Armstrong  &  Overton,  1977). 
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Table  25.  Levene’s  Test  Results 


dfl 

Sig. 

FEAR 

.526 

2 

347 

.591 

CO_AUTH 

.014 

2 

347 

.986 

PALT_P 

1.777 

2 

347 

.171 

TEE 

.204 

2 

347 

.815 

RCI 

2.060 

2 

347 

.129 

BS 

.686 

2 

347 

CP 

.279 

2 

347 

.757 

SSMA 

.741 

2 

347 

.477 

PR 

2.526 

2 

347 

Table  26.  Univariate  F-Tests  for  Non-Response  Bias 


Type  III  Sum  of 

Dependent  Variable  Squares  Df 

Mean  Square 

P 

Sig. 

PEAR 

150.652" 

2 

75.326 

1.246 

.289 

CO_AUTH 

3.320'’ 

2 

1.660 

.241 

.786 

PAET_P 

75.540" 

2 

37.770 

1.873 

.155 

TEE 

2.104'' 

2 

1.052 

.060 

.942 

RCI 

2.474" 

2 

1.237 

.290 

.748 

BS 

6.383^ 

2 

3.192 

1.117 

.328 

CP 

269.81U 

2 

134.906 

2.035 

.132 

SSMA 

4.758'’ 

2 

2.379 

.053 

.948 

PR 

4.769' 

2 

2.384 

.406 

.667 

Cl 

198.387j 

2 

99.193 

.957 

.385 

In  summary,  nine  variables  were  tested  for  the  presence  of  non-response  bias  using  the 
extrapolation  method  and  multivariate  analysis  of  variance.  The  results  of  testing  did  not  provide 
statistical  evidence  to  reject  the  null  hypotheses  that  the  means  of  the  first,  second,  and  third 
wave  respondents’  variables  were  equal.  It  can  be  assumed  that  non-respondents  missing  from 
the  sample  are  random  and  that  further  analysis  will  not  be  biased  because  of  non-respondents 
(Muir,  2010,  p.  60). 
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E.  SAMPLE  SIZE 


To  ensure  a  valid  measure  fit  of  relationships,  reduce  any  sample  errors,  and  reduce  the 
detrimental  effects  of  non-normality,  most  researchers  suggest  using  a  sample  size  of  200  (Hair 
et  al.,  2010).  However,  it  was  also  suggested  that  a  minimum  sample  size  of  100  could  be  used 
(Smith  &  Langfield-Smith,  2004).  The  effects  of  the  sample  size  can  be  observed  most  in  the 
statistical  power  of  significance  testing,  and  more  importantly  in  the  generalizability  of  the 
results. 

According  to  Hair  et  al.  (2010),  sample  size  must  have  the  following  for  a  factor  analysis 

design: 

•  The  sample  must  have  more  observations  than  variables. 

•  The  minimum  absolute  sample  size  should  be  50  observations. 

•  Researchers  should  strive  to  maximize  the  number  of  observations  per  variable, 
with  a  desired  ratio  of  5  observations  per  variable. 

As  a  general  rule,  the  ratio  of  observations  to  independent  variables  should  never  be 
below  5:1.  The  desired  amount  would  be  a  ratio  of  15  to  20:1.  If  this  ratio  can  be  achieved,  the 
results  should  be  generalizable.  This  study  contains  15  variables  with  350  observations,  which 
yields  a  ratio  of  23.33:1.  This  ratio  more  than  satisfies  the  referenced  generalizability  guidelines. 

F.  EXPLORATORY  FACTOR  ANALYSIS 

As  a  preliminary  step  to  establishing  construct  validity,  scales  were  purified  using 
exploratory  factor  analysis  in  SPSS  Version  22.  The  factor  loadings  matrix  produced  is  presented 
in  Table  27.  Factor  analysis  is  a  technique  whose  primary  purpose  is  to  define  the  underlying 
structure  among  variables.  It  is  primarily  used  to  analyze  the  structure  of  the  correlations  among 
a  large  number  of  variables  by  identifying  sets  of  variables  that  have  high  influence  on  each 
other,  known  as  factors  (Hair  et  al.,  2010). 

Factor  loading  is  the  correlation  of  a  variable  and  the  factor,  while  the  squared  loading  is 
the  amount  of  the  variable’s  total  variance  accounted  for  by  the  factor  (Hair  et  al.,  2010).  When 
assessing  the  significance  of  factor  loading,  the  first  guideline  is  to  do  a  preliminary  examination 
of  the  factor  loadings.  The  general  rules  for  practical  significance  are: 
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•  factor  loadings  ranging  from  +  .30  to  +.40  are  the  minimal  level  for  interpretation 
of  structure, 

•  factor  loading  +  .50  are  considered  practically  significant,  and 

•  factor  loadings  exceeding  .70  are  indicative  of  well-defined  structure. 
Interpretation  of  the  factor  matrix  starts  with  the  first  variable  of  the  first  factor  and 

moved  horizontally  from  left  to  right.  The  goal  is  to  identify  the  highest  loading  of  that  factor. 
This  is  done  for  each  variable  until  all  have  been  examined.  Variables  with  insignificant  loadings 
are  then  discarded.  When  a  variable  is  found  to  have  more  than  one  significant  loading,  it  is 
called  a  cross  loading  variable.  Cross-loaded  variables  should  be  minimized  if  possible.  Once  all 
significant  loadings  have  been  identified,  reassessment  is  done,  removing  insignificant  and 
highly  cross-loaded  variable. 

Nine  variables  were  discarded  because  of  insignificant  loadings  or  high  cross  loadings. 
Four  of  the  nine  items  were  from  established  scales.  All  other  variables  retained  displayed 
significance  factors  greater  than  the  generally  accepted  threshold  of  .5.  Only  two  items  in  the 
contracting  officer  authority  construct  contained  significance  loadings  of  less  than  .6.  Five  items 
displayed  cross-loadings  below  the  0.300  limit  recommended  by  Hair  et  al.  (2010),  but  were 
maintained  for  summation. 
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Table  27. 


Component 

1 

2 

3 

4 

FEARl 

.854 

EEAR2 

.885 

EEAR3 

.837 

EEAR5 

.863 

EEAR6 

COAUTHl 

COAUTH2 

COAUTH3 

COAUTH4 

COAUTH6 

PALTPl 

PALTP2 

PALTP3 

TEE2 

TEE3 

TEE6 

RCI2 

.786 

-.906 

RCI3 

-.901 

RCI4 

-.831 

CPI 

.910 

CP2 

.928 

CP3 

.877 

CP4 

.828 

CP5 

.896 

CP6 

.807 

CP7 

SSMAl 

.886 

.866 

SSMA2 

.837 

SSMA3 

.764 

SSMA4 

.733 

SSMA5 

.762 

SSMA6 

BS3 

.264 

.805 

BS4 

.214 

BS2 

.223 

Extraction  Method:  Principal  Component  Analysis. 
Rotation  Method:  Oblimin  with  Kaiser  Normalization, 
a.  Rotation  converged  in  6  iterations. 


Factor  Loadings 


.862 

.907 

.819 

.571 

.586 


-.213 


.232 


.792 

.827 

.877 


.889 

.892 

.609 


.734 

.662 

.750 
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Summated  scales  combine  several  like  variables  into  a  single  composite  measure.  This  is 
done  by  combining  the  variables  that  are  highly  loaded  on  a  factor.  Summated  scales  have  two 
specific  benefits:  They  provide  a  means  of  overcoming  to  some  extent  the  measurement  error 
inherent  in  all  measured  variables  and  its  ability  to  represent  multiple  aspects  of  a  concept  in  a 
single  measure  (Hair  et  ah,  2010).  Descriptions  for  the  summated  scales  can  be  seen  in  Table  28. 
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Table  28.  Construct  Descriptives 


Summated  Construct 

N 

Min 

Max 

Mean 

Std.  Deviation  Skewness 

Kurtosis 

FEAR 

350 

5 

35 

19.71 

7.782 

-.107 

-.798 

Contracting  Officer  Authority  (CO_AUTH) 

350 

5 

35 

25.05 

6.021 

-.820 

.447 

PALT  Planned  (PALT_P) 

350 

3 

20 

10.81 

4.502 

-.092 

-.921 

Tech  Eval  Effectiveness  (TEE) 

350 

3 

21 

10.34 

4.177 

.315 

-.641 

Requirement  Criticality  (RCI) 

350 

4 

21 

17.15 

3.008 

-.847 

.856 

Buyer  Satisfaction  (BS) 

350 

6 

21 

13.99 

1.691 

-.483 

2.861 

Contractor  Performance  (CP) 

350 

7 

49 

33.08 

8.166 

-.201 

.282 

Source  Sel.  Method  Appropriateness  (SSMA) 

350 

6 

42 

33.06 

6.655 

-1.174 

1.585 
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Reliability  is  also  a  consideration  in  assessing  the  degree  of  consistency  between  multiple 
measures  of  a  variable  (Hair  et  al.,  2010).  Reliability  is  a  necessary,  but  not  sufficient,  aspect  of 
construct  validity.  One  measure  of  construct  reliability  is  the  Cronbach’s  Alpha,  which  is  also 
known  as  the  reliability  coefficient.  The  most  widely  used  lower  limit  for  Cronbach’s  Alpha  is 
.70,  although  .60  may  be  acceptable  in  exploratory  research  (Nunnally,  1978).  Table  29  shows 
the  reliability  coefficients  for  summated  scales  resulting  from  exploratory  factor  analysis. 


Table  29.  Cronbach’s  Alpha  Scores 


Construct 

Cronbach’s 

Alpha 

FEAR 

.904 

CO_AUTH 

.826 

PALT_P 

.805 

TEE 

.758 

RCI 

.856 

BS 

.880 

CP 

.963 

SSMA 

.901 

G.  HYPOTHESIZED  MODELS 

The  final  step  of  analysis  was  the  regression  analysis  of  the  hypothesized  regression 
models  in  SPSS  version  22.  Three  methods  of  regression  were  used  to  analyze  various 
hypothesized  relationships  between  constructs.  Multiple  regression  was  used  to  measure  the  left 
side  of  the  model,  which  analyzed  the  factors  that  contributed  to  protest  fear.  Simple  linear  and 
logistic  regressions  were  performed  on  the  right  side  of  the  model  to  measure  the  resultant 
effects  of  protest  fear. 

2  2 

The  model  summary  of  each  regression  featuring  the  R  and  adjusted  R  values  was 
analyzed  to  assess  the  total  variation  explained  in  the  dependent  variable  by  the  independent 
variable(s).  The  significance  value  in  the  Analysis  of  Variation  (ANOVA)  table  was  then 
examined  to  determine  the  overall  validity  of  the  model.  To  test  the  validity  of  the  model,  the 
following  hypotheses  are  assumed: 

•  Ho:Pi  =  P2  =  -  =  Pk  =  0 

•  Hi:  At  least  one  Pi  is  not  equal  to  0 
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If  the  null  hypothesis  is  true,  none  of  the  independent  variable  is  linearly  related  to  the 
dependent  variable,  and  the  model  is  invalid  or  not  statistically  significant.  A  minimum  of  one 
i3  ^  must  be  not  equal  to  0  for  the  model  to  have  validity.  The  significance  test  in  the  ANOVA 
table  combines  multiple  Mests  into  a  single  test  (Keller,  2008). 

The  Coefficients  table  is  the  last  part  of  the  regression  analysis.  The  Unstandardized 
Coefficients  ((B)  describe  the  relationship  between  each  independent  variable  and  the  dependent 
variable.  The  significance  value  for  each  independent  variable  was  then  analyzed  to  determine 
statistical  significance  of  each  independent  variable.  All  tests  used  an  alpha  (a)  value  of  0.05, 
meaning  that  any  significance  value  greater  than  a  was  deemed  not  significant.  Alpha  values 
between  0.05  and  0.10  were  considered  marginally  significant. 

1.  Antecedents  of  Fear  Results 

Multiple  regression  analysis  was  performed  on  the  summated  constructs  that  measured 
the  relationships  identified  in  the  left  side  of  the  model.  A  visual  depiction  of  the  model  is  shown 
in  Figure  2.  The  model  used  Planned  PALT  (PALT_P),  Dollar  Value  (D_VAL),  Competency 
(Cl  and  C2),  Requirement  Criticality /Importance  (RCI),  and  Protest  Risk  (PR)  as  independent 
variables  measuring  the  variance  in  the  dependent  variable  Fear  (FEAR). 

The  statistical  significance  of  the  model  was  then  examined  in  the  ANOVA  table.  The 
significance  of  the  model  was  reported  at  .000  using  an  a  value  of  0.05.  Since  the  significance  is 
less  than  a,  we  can  reject  the  null  hypothesis  that  there  is  no  linear  relationship  between  FEAR 
and  the  independent  variables  in  the  model.  The  ANOVA  table  is  shown  in  Table  31. 

The  Model  Summary  table  for  this  model  can  be  seen  in  Table  30.  The  Correlation 
Coefficient  (R)  was  .361.  The  Coefficient  of  Determination  (R  )  was  .131  with  an  adjusted  R  of 
.115.  The  R  value  means  that  13.1  percent  of  the  variation  in  the  dependent  variable  is  explained 
by  the  independent  variables.  The  proportion  left  unexplained,  which  is  called  the  coefficient  of 
alienation,  is  .869.  Sirkin  (2006,  p.  474)  noted  that  there  are  “many  real  world  factors  that 
mitigate  linearity  in  social  sciences.  It  would  be  rare  indeed  to  see  coefficients  of  determination 
approaching  1.0  (or  even  .9  or.8)  in  social  or  behavioral  research.”  He  then  added  that  “R  is  a 
useful  concept  in  statistics,  it  is  less  useful  in  the  real  world  of  data  analysis  in  determining  what 
is  or  is  not  a  “good”  linear  relationship.  Thus  statistical  significance  is  often  used  to  demonstrate 
“good”  relationships”  (Sirken,  2006,  p.  457).  While  the  R  values  are  high  enough  to  substantiate 
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this  model,  the  model  also  suggests  that  there  are  some  unobserved  factors  that  may  explain 
more  of  the  variance  in  the  FEAR  of  protest  variable. 

Finally,  the  Coefficients  Table  was  examined  for  the  (B  values  and  significance  of  each 
independent  variable.  Five  out  of  six  independent  variables  had  significance  values  less  than  .05, 
with  the  variable  Years  of  Experience  having  a  significance  value  of  .379.  PAFT  Planned  had  a 
(B  value  of  -.259  and  a  significance  of  .004.  This  shows  that  as  PAFT  is  increased,  the  level  of 
FEAR  decreases  by  .259  (on  a  seven-point  scale)  assuming  all  other  variables  remain  constant. 
Dollar  Value  had  a  (B  value  of  .721  with  a  significance  value  of  .003.  This  shows  that  as  Dollar 
Value  increases  by  one  unit,  FEAR  also  increases  by  .721  (on  a  seven-point  scale)  assuming  all 
other  variables  remain  constant.  It  is  important  to  note  that  the  Dollar  Value  construct  does  not 
measure  the  actual  dollar  value  of  the  acquisition.  The  actual  dollar  values  were  not  normally 
distributed  and  therefore  inappropriate  for  regression  analysis.  Requirement  Criticality  had  a  (B 
value  of  .431  with  a  significance  value  of  .002.  This  shows  that  as  Requirement  Criticality 
increases  by  one  unit,  FEAR  also  increases  by  .43 1  (on  a  seven-point  scale)  assuming  all  other 
variables  remain  constant.  Protest  Risk  had  a  (B  value  of  .374  with  a  significance  value  of  .029. 
This  shows  that  as  Protest  Risk  increases  by  one  unit,  FEAR  also  increases  by  .374  (on  a  seven- 
point  scale)  assuming  all  other  variables  remain  constant.  Years  of  Experience  had  a  (B  value  of 
.037  but  was  insignificant.  The  number  of  Source  Selections  had  a  (B  value  of  -.267  with  a 
significance  value  of  .01 1.  This  shows  that  as  the  Number  of  Source  Selections  increases  by  one 
unit,  FEAR  decreases  by  .267  (on  a  seven-point  scale)  assuming  all  other  variables  remain 
constant.  Results  of  this  test  can  be  found  in  Table  32.  A  visual  depiction  of  all  relationships  is 
shown  in  Figure  2. 


Table  30.  Model  Summary 


m 

R 

R  Square 

Adjusted  R 

Square 

Std.  Error  of 
the  Estimate 

.131 

a.  Predictors:  (Constant),  Number  of  Source  Selections, 
Protest  Risk,  Dollar  Value,  Requirement  Criticality,  PAFT 
Planned,  Years  of  Experience 
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Table  31.  ANOVA‘ 


Model 

df 

Mean  Square 

F 

Sig. 

1  Regression 

2759.668 

6 

459.945 

8.587 

Residual 

18373.021 

343 

53.566 

Total 

21132.689 

349 

a.  Dependent  Variable:  FEAR 

b.  Predictors:  (Constant),  Number  of  Source  Selections,  Protest  Risk,  Dollar  Value, 
Requirement  Criticality,  PALT  Planned,  Years  of  Experience 


Table  32.  Coefficients 


Unstandardized 

Coefficients 

Standardized 

Coefficients 

Construct 

B 

Std.  Error 

Beta 

t 

Sig. 

(Constant) 

10.995 

2.852 

3.856 

.000 

PAET  Planned 

-.259 

.091 

-2.862 

Dollar  Value 

.721 

.238 

.155 

Requirement  Criticality 

.431 

.135 

.167 

3.183 

Protest  Risk 

.374 

.171 

.116 

Years  of  Experience 

.042 

.881 

.379 

Number  of  Source  Selections 

-.267 

.104 

-.142 

-2.558 

a.  Dependent  Variable:  FEAR 
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2.  Consequences  of  Fear  Results 

One  of  the  objectives  of  this  research  is  to  determine  the  consequences  of  protest  fear. 
Multiple  variables  were  used  to  gauge  the  lengths  that  contracting  personnel  would  take  to  avoid 
a  protest.  Some  constructs,  such  as  the  amount  of  awarded  contracts,  were  very  straightforward 
in  ascertaining  whether  a  problem  exists. 

The  right  side  of  the  model  was  then  tested  through  a  combination  of  simple  linear 
regressions  and  logistic  regressions  in  SPSS  Version  22.  Key  statistics  of  these  tests  can  be  found 
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in  Table  33.  Logistic  regression,  being  well  suited  for  analyzing  dichotomous  outcomes,  is 
finding  increased  popularity  in  social  science  research  (Peng  &  So,  2002).  It  is  the  appropriate 
statistical  technique  for  performing  analysis  when  the  dependent  variable  is  categorical.  Hair  et 
al.  (2011)  stated  that  Logistic  Regression  is  best  suited  to  deal  with  two  research  objectives: 
identifying  the  independent  variables  that  impact  group  membership  in  the  dependent  variable, 
and  establishing  classification  systems  based  on  the  logistic  model  for  determining  group 
membership.  Logistic  regression  estimates  the  probability  that  a  particular  outcome  will  occur. 
The  relationship  between  independent  and  dependent  variable  is  expressed  in  the  form  of  an  odds 
ratio,  which  expresses  the  probability.  If  the  ratio  is  positive,  “an  increase  in  that  independent 
variable  will  result  in  an  increase  in  the  probability  of  the  event”  (Keller,  2008,  p.  742). 

Linear  regression  was  performed  to  analyze  the  relationship  between  the  FEAR  of  protest 
and  technical  evaluation  effectiveness  (TEE),  with  EEAR  being  the  independent  variable  and 
TEE  being  the  dependent  variable.  TEE  was  measured  using  reverse-coded  survey  questions 
(items);  thus,  the  inverse  of  the  given  responses  was  used  in  the  test.  There  was  a  significant 
relationship  (.000)  between  the  two  variables.  The  (B  coefficient  was  -.136,  which  was  in  the 
hypothesized  direction. 

The  second  test  performed  was  a  linear  regression  analyzing  the  relationship  between 
EEAR  of  protest  and  perceived  source  selection  method  appropriateness  (SSMA),  using  EEAR 
as  independent  variable  and  SSMA  as  dependent  variable.  The  test  failed  to  reject  the  null 
hypothesis  (.275)  that  there  was  no  linear  relationship  between  EEAR  and  SSMA. 

Since  this  analysis  dealt  with  the  consequences  of  protest  fear,  some  aspects  of  different 
constructs  were  transformed  into  binary-coded  variables.  SSME  and  SSME  were  used  to 
measure  the  difference  between  planned  and  actual  procurement  lead-time  (PAET).  SSME^ 
asked  how  many  days  were  planned  for  the  Source  Selection  in  question,  and  SSME^  asked  how 
many  days  the  Source  Selection  actually  consumed.  To  understand  the  effects  of  the  independent 
variable  EEAR  of  protest  on  Actual  PAET,  source  selections  that  took  longer  than  planned  were 
coded  as  1  and  those  that  were  less  or  the  same  were  coded  as  0.  This  model  was  significant 
(.000),  but  the  Odds  Ratio  of  1.051  indicates  slightly  higher  than  1  to  1  odds  of  having  increased 
Actual  PAET  due  to  protest  fear. 

SSME^  and  SSME"^  were  used  to  measure  resources  consumed  to  conduct  the  source 
selection.  SSME^  asked  how  many  personnel  were  on  the  Source  Selection  team  and  SSME"^ 
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asked  how  many  there  should  have  been  absent  a  fear  of  protest.  Three  tests  were  run  using 
FEAR  of  protest  as  the  independent  variable.  The  first  was  a  linear  regression  using  SSMF  as 
the  dependent  variable.  This  test  was  significant  (.000)  with  a  (B  coefficient  of  .010.  Thus,  as  the 
fear  of  protest  increases,  so  does  the  number  of  personnel  assigned  to  a  source  selection.  The 
next  test  used  a  binary  variable  for  FEAR  of  protest  labeled  FEAR  HFFO,  where  a  respondent 
whose  summated  FEAR  rating  was  above  average  was  coded  as  1,  and  all  others  were  coded  as 
0.  SSMF  again  was  the  dependent  variable.  Again,  a  linear  regression  was  performed  since  the 
dependent  variable  was  not  binary.  This  model  was  also  significant  (.000),  but  displayed  a  higher 
P  coefficienct  of  .149.  The  last  test  was  a  logistic  regression  using  the  difference  between  SSMF^ 
and  SSMF*^.  Those  responses  that  had  more  people  on  the  team  than  the  respondent  thought 
should  be  (absent  a  fear  of  protest)  were  coded  as  1;  all  others  were  coded  as  0.  This  test  was  not 
significant  (.379). 

A  series  of  logistic  regressions  were  then  run  on  the  appropriateness  of  two  types  of 
source  selections — FPTA  and  Full  Trade-Off — using  responses  from  the  variables  SSMF^  and 

o 

SSMF  .  The  first  step  in  the  test  was  to  filter  in  only  the  respondents  that  used  the  particular 
Source  Selection  Method  (i.e.,  FPTA  or  full  trade-off).  A  dummy  variable  was  then  created  to 
ascertain  how  appropriate  the  respondent  thought  the  use  of  the  method  was.  The  questions  used 
a  seven-point  Fikert-type  scale  with  1  being  “Completely  Inappropriate”  and  7  being 
“Completely  Appropriate.”  Respondents  that  answered  the  questions  with  scores  of  3  or  less 
were  coded  as  1,  and  all  others  were  coded  as  0.  Of  note,  there  were  13  respondents  out  of  133 
that  used  FPTA  even  though  they  felt  it  was  to  some  degree  inappropriate.  10  respondents  out  of 
174  that  used  Full  Trade-off  even  though  they  felt  it  was  to  some  degree  inappropriate.  This 
inappropriate  use  totals  7.49  percent  of  the  307  respondents  that  used  either  FPTA  or  trade-off  as 
a  source  selection  method.  Both  tests  used  the  same  binary  FEAR  HI/FO  independent  variable  in 
the  aforementioned  tests.  The  logistic  regression  test  to  measure  the  relationship  with  the  use  of 
the  Full  Trade-off  inappropriateness  was  not  significant  (.181).  The  test  using  FPTA 
inappropriateness  was  significant  (.010)  with  an  Odds  Ratio  of  4.673. 

Three  tests  were  then  performed  to  measure  the  relationship  between  the  FEAR  of  protest 
and  the  use  of  discussions  using  SSMF^,  SSMF^°,  and  SSMF'^.  The  first  test  used  the  binary 
FEAR  HI/FO  independent  variable  to  measure  the  level  of  satisfaction  with  the  degree  of 
discussions,  using  SSMF^  as  the  dependent  variable.  SSMF^  uses  a  seven-point  Fikert-type  with 
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1  being  “Completely  Satisfied”  and  7  being  “Completely  Dissatisfied.”  The  test  was  signifieant 
(.037)  with  a  (B  eoefficient  of  .382.  The  test  signifies  that  if  FEAR  of  protest  is  high,  the  level  of 
dissatisfaetion  with  diseussions  rises  slightly.  The  next  test  used  the  FEAR  HEEO  binary 
independent  variable  and  SSMF^°  as  the  dependent  variable.  SSME^*^  examined  how  mueh  the 
level  of  diseussions  would  ehange  if  there  were  no  ability  to  protest.  It  used  a  seven-point  Eikert- 
type  seale  with  1  representing  “No  Difference”  and  7  representing  a  “Substantial  Difference.” 
This  test  was  also  significant  (.000)  with  a  (3  of  1.289.  The  test  signifies  that  respondents  with 
high  FEAR  of  protest  would  change  the  manner  in  which  they  conducted  discussions  if  there 
were  no  threat  of  protest.  The  last  test  was  a  Eogistic  Regression  performed  on  FEAR  as  the 
independent  variable  and  SSMF  .  SSMF  asked  whether  discussions  had  been  conducted  or 
not,  and  was  binary  coded  where  1  represented  Yes  and  0  represented  No.  This  test  was  also 
significant  (.037),  but  the  Odds  Ratio  of  1.030  shows  only  a  slightly  better  than  to  1  odds  of 
FEAR  having  an  effect  on  whether  discussions  are  conducted.  Hence,  fear  of  protest  tends  to 
slightly  increase  the  odds  that  discussions  will  occur. 

The  next  test  was  a  linear  regression  performed  to  analyze  the  appropriateness  of  oral 
presentations.  The  test  used  FEAR  as  an  independent  variable  and  SSMF^^  as  the  dependent 
variable.  This  test  did  not  reject  the  null  hypothesis  (.261)  that  there  is  no  linear  relationship 
between  the  two  variables. 

The  Awarded  Contracts  variable  (AC)  measured  the  difference  between  how  many 
contracts  were  planned  and  how  many  were  actually  awarded.  The  AC  variable  was  binary 
coded  where  respondents  that  saw  more  contracts  awarded  than  planned  was  coded  as  1  and  all 
others  were  0.  There  were  19  instances  where  more  contracts  were  awarded  than  originally 
planned,  representing  5.42  percent  of  respondents.  However,  this  test  was  insignificant  (.350) 
and  no  further  analysis  of  this  relationship  was  done. 

Transaction  Costs  consists  of  two  separate  variables  comprised  of  questions  measuring 
the  number  of  iterations  of  key  source  selection  documents  (TCI)  and  personnel  costs  of  those 
associated  with  the  source  selection  (TCP).  Descriptive  statistics  of  the  TCI  variables  are  shown 
in  Table  33.  TCP  was  calculated  by  asking  respondents  the  number,  rank,  and  fraction  of  one 
year’s  time  spent  on  the  source  selection  by  role.  The  average  2013  General  Schedule  annual  pay 
was  used  to  calculate  the  cost  for  each  role,  and  then  summed  for  each  source  selection  into  the 
TCP  dependent  variable.  The  values  ranged  from  a  low  of  $7,000.00  to  a  maximum  of 
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$3,551,944.33.  The  average  cost  per  source  selection  was  $235,236.34,  with  a  standard  deviation 
of  $291,620.05.  Linear  regression  was  used  to  test  the  relationship  between  the  independent 
variable,  FEAR  of  protest,  and  TCP.  The  test  was  significant  (.018)  but  displayed  a  very  low  (B  of 
.009.  While  there  was  a  statistically  significant  relationship  between  these  two  variables,  the 
effect  size  is  minute. 

The  next  series  of  tests  were  on  the  TCI  variables.  Normality  was  achieved  on  only  four 
types  of  source  selection  documents  of  many  types  of  documents  measured — Source  Selection 
Plan,  Debriefing  Script,  Color/Rating  Chart,  and  Evaluation/Briefing  Chart  to  the  Source 
Selection  Authority.  The  first  test  measured  the  relationship  between  the  independent  variable, 
EEAR,  and  TCI"^,  which  represents  the  number  of  iterations  of  the  Source  Selection  Plan.  This 
test  was  shown  to  be  significant  (.000),  and  had  a  (B  of  .023.  The  second  test  measured  the 
relationship  between  the  independent  variable,  PEAR,  and  TCI^,  which  represents  the  number  of 
iterations  of  the  Debriefing  Script.  This  test  was  also  shown  to  be  significant  (.000),  and  had  a  (B 
of  .028.  The  third  test  measured  the  relationship  between  the  independent  variable,  PEAR,  and 
TCI^,  which  represents  the  number  of  iterations  of  the  Color/Rating  Chart.  This  test  was  shown 
to  be  significant  (.000),  and  had  a  (B  of  .020.  Pinally,  the  last  test  measured  the  relationship 
between  the  independent  variable,  PEAR,  and  TCI^**,  which  represents  the  number  of  iterations 
of  the  Evaluation/Briefing  Chart  to  the  Source  Selection  Authority.  This  test  was  shown  to  be 
significant  (.000),  and  had  a  (B  of  .018.  While  there  is  a  significant  relationship  between  PEAR  of 
protest  and  the  number  of  changes  to  each  of  these  documents,  the  magnitudes  of  the 
relationships  were  small. 
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Table  33.  Document  Iteration  Descriptive 


Document 

N 

Min 

Max 

Mean 

Std.  Deviation 

Source  selection  decision  document 

350 

.00 

12.00 

2.6429 

1.80604 

Comp,  assessment/proposal  analysis  report 

350 

.00 

10.00 

2.4743 

1.66307 

Evaluation  notices 

350 

.00 

21.00 

2.3314 

2.55796 

Source  selection  plan 

350 

.00 

12.00 

2.7514 

2.02093 

Debriefing  scripts 

350 

.00 

15.00 

1.9657 

1.98099 

Technical  evaluations 

350 

.00 

17.00 

3.4143 

2.38244 

Past  performance  evaluations 

350 

.00 

20.00 

2.7486 

2.62666 

Cost/Price  analysis 

350 

.00 

17.00 

2.3086 

1.87937 

Color/rating  chart 

350 

.00 

11.00 

1.3257 

1.60128 

Evaluation  briefing  charts  for  reviewers  and  SSA 

350 

.00 

10.00 

1.7257 

1.61624 

The  last  hypothesized  consequence  of  FEAR  to  be  tested  was  the  effect  on  contracting 
officer  authority.  Two  tests  were  performed  using  CO  AUTH  as  the  dependent  variable.  The  first 
test  used  the  binary  FEAR  HI/EO  and  the  second  test  used  a  continuous  measure  of  FEAR  (on  a 
one  to  seven  scale)  as  the  independent  variable.  The  first  test  had  a  significance  of  .914,  which  is 
not  significant.  The  second  test  had  a  significance  of  .093  which  is  considered  marginally 
significant  using  an  a  value  of  .10.  The  P  coefficient  was  -.070,  which  was  in  the  direction 
hypothesized. 

The  object  of  the  final  set  of  tests  was  to  analyze  hypothesized  relationships  between  the 
technical  evaluations’  effectiveness  (TEE)  and  perceived  source  selection  method 
appropriateness  on  contractor  performance  (CP).  Multiple  regression  was  used  to  analyze  this 
relationship.  The  construct  TEE  and  SSMA  were  used  as  the  independent  variables  and  CP  was 
used  as  the  dependent  variable.  The  overall  test  was  significant  (.000).  TEE  was  significant  as  an 
independent  variable  (.001)  and  had  a  p  of  .327.  SSMA  was  also  significant  as  an  independent 
variable  (.000)  and  had  a  p  of  .328.  This  test  showed  significant  relationships  between  the  TEE 
and  CP,  and  between  SSMA  and  CP.  The  ps  indicated  moderate  effect  sizes. 

Next,  two  exploratory  tests  were  performed  on  the  inappropriate  use  of  EPTA  as  the 
independent  variable  and  contractor  performance  as  the  dependent  variable.  Only  the  133 
responses  that  used  EPTA  were  included  in  these  tests.  For  the  first  test,  a  binary  variable  was 
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used,  in  which  respondents  that  felt  LPTA  was  anything  other  than  appropriate  (4  or  less)  were 
coded  as  1  and  all  others  were  0.  The  test  showed  a  significant  relationship  (.010)  between  the 
variables,  and  a  (B  of  -4.355  indicating  that  CP  decreases  significantly  for  those  respondents  that 
felt  the  use  of  LPTA  was  other  than  appropriate.  The  second  test  used  the  continuous  LPTA 
appropriateness  scores  where  1  represents  “completely  inappropriate”  and  7  represents 
“completely  appropriate.”  CP  was  the  dependent  variable.  This  test  was  also  significant  (.006) 
with  a  (3  of  1.237.  This  shows  that  as  LPTA  appropriateness  increases  by  one  unit,  CP  also 
increases  by  1.237  (on  a  seven-point  scale). 

The  final  test  was  a  linear  regression  to  measure  the  relationship  between  contractor 
performance  and  buyer  satisfaction.  The  construct  CP  was  used  as  the  independent  variable  and 
BS  was  used  as  the  dependent  variable.  The  test  showed  a  significant  (.000)  positive  relationship 
between  the  two  variables;  the  magnitude  of  the  (3  was. 285.  A  summary  table  of  all  key 
regression  data  can  be  found  in  Table  34.  A  visual  depiction  of  all  relationships  associated  with 
the  right  side  of  the  model  can  be  found  in  Figure  3. 
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Table  34.  Key  Statistics  from  Right-Side  (Consequences)  Regressions 


Independent  Variable 

Dependent  Variable 

N  R^ 

AdjR^ 

D.F. 

Sig. 

P 

(3  Sig. 

FEAR 

TEE 

350  .064 

.061 

349 

.000 

-.136 

.000 

FEAR 

PALT  Actual** 

350  .047 

- 

349 

.000 

1.051* 

.001 

FEAR 

SSMA 

350  .003 

.001 

349 

.275 

-.050 

.275 

FEAR  HI/EO** 

SSMF®**  (EPTA  Appropriateness) 

133  .048 

- 

132 

.010 

4.673* 

.015 

FEAR  HI/EO** 

SSMF^**(T/0  Appropriateness) 

174  .010 

- 

173 

.181 

.416* 

.187 

FEAR  HI/EO** 

SSMF^  (Satisfaction  w/  discussions) 

350  .012 

.010 

349 

.037 

.382 

.037 

FEAR  HI/EO** 

SSMF'°  (Hypothetical  change) 

350  .100 

.097 

349 

.000 

1.289 

.000 

FEAR 

SSMF'^**  (Discussions  Held  Y/N) 

350  .012 

- 

349 

.037 

1.030* 

.038 

FEAR 

SSMF'^  (Oral  Pres.  Appropriateness) 

350  .004 

.001 

349 

.261 

-.016 

.261 

FEAR 

SSMF^  (Resources) 

350  .077 

.074 

349 

.000 

.010 

.000 

FEAR  HI/EO** 

SSMF^  (Resources) 

350  .066 

.063 

349 

.000 

.149 

.000 

FEAR 

SSMF^**  (More  Resources  than  needed) 

350  .002 

- 

349 

.378 

1.013* 

.379 

FEAR 

TCI 

350  .101 

.099 

349 

.000 

.014 

.000 

FEAR 

TCC  (Source  Selection  Plan) 

350  .084 

.082 

349 

.000 

.023 

.000 

FEAR 

TCC  (Debriefing  Scripts) 

350  .078 

.076 

349 

.000 

.028 

.000 

FEAR 

TCC  (Color/Rating  Chart) 

350  .038 

.035 

349 

.000 

.020 

.000 

FEAR 

TCl’°  (Evaluation/Briefing  Chart  for  SSA) 

350  .038 

.035 

349 

.000 

.018 

.000 

FEAR 

TCP 

270  .021 

.017 

269 

.018 

.009 

.018 

FEAR 

AC** 

350  .002 

- 

349 

.350 

1.029* 

.355 

FEAR 

CO  AUTH 

350  .008 

.005 

349 

.093 

-.070 

.093 

FEAR  HI/EO** 

CO  AUTH 

350  .000 

-.003 

349 

.914 

-.070 

.914 

TEE*** 

CP 

350  .123 

.118 

349 

.000 

.327 

.001 

SSMA*** 

CP 

.328 

.000 

SSMF®**(LPTA  Appr) 

CP 

133  .049 

.042 

132 

.010 

-4.355 

.010 
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SSMF® 

CP 

133  .057 

.049 

132 

.006 

1.237 

.006 

CP 

BS 

350  .442 

.441 

349 

.000 

.285 

.000 

Exp  (B  from  Logistic  Regression 
**Binary 

***Multiple  Regression 
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Tech  Eval  Effectiveness 


Figure  3.  Right  Side  of  Hypothesized  Model 
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Figure  4.  Statistically  Significant  Model 
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H.  CONCLUSION 


In  summary,  this  chapter  reported  the  results  of  the  linear,  multiple,  and  logistic 
regressions  performed  on  the  350  usable  survey  responses.  Firstly,  the  350  responses 
were  analyzed  to  verify  that  the  data  was  appropriate  for  being  used  in  regression  analysis 
(i.e.,  that  the  assumptions  of  regression  were  satisfied).  Exploratory  Factor  Analysis  was 
then  performed  to  purify  scales,  in  which  35  items  were  retained  for  eight  constructs, 
with  all  constructs  being  measured  by  no  fewer  than  three  items  and  exhibiting  sufficient 
reliability  coefficients  (Muir,  2010).  Finally,  the  hypothesized  model  was  analyzed  using 
regression  analysis.  Several  relationships  were  shown  to  be  statistically  significant,  which 
supported  12  of  14  proposed  hypotheses.  Fastly,  the  hypothesized  model  was  presented 
with  only  the  statistically  significant  relationships  shown  (Figure  4).  For  ease  of 
interpretation,  a  summary  table  containing  the  significance  level  of  each  hypothesis  can 
be  found  in  Table  35. 
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Table  35.  Summary  of  Hypothesis  Testing 

Hypothesis 

Supported? 

HI 

Insufficient  PALT  has  a  direct  positive  relationship  on  the  level  of  fear  of  protest. 

Yes 

H2 

The  greater  a  contracting  officer’s  Competence  level,  the  lower  the  level  of  fear 
of  protest. 

Yes 

H3 

The  level  of  requirement  criticality /importance  has  a  positive  impact  on  the  level 
of  fear  of  protest. 

Yes 

H4 

As  protest  risk  increases,  the  level  of  fear  of  protest  increases. 

Yes 

H5 

The  level  of  fear  of  protest  has  a  negative  effect  on  the  technical  evaluation 
effectiveness. 

Yes 

H6 

There  is  a  direct  positive  relationship  between  fear  of  protest  and  the  Source 
Selection  method  fit/  appropriateness. 

Partially 

H7 

There  is  a  positive  correlation  between  the  fit/appropriateness  of  a  source 
selection  and  contractor  performance. 

Partially 

H8 

There  is  a  positive  relationship  between  technical  evaluation  effectiveness  and 
contractor  performance. 

Yes 

H9 

There  is  a  direct  positive  relationship  between  contractor’s  performance  based  on 
the  source  selection  method  used  and  the  Buyer’s  satisfaction. 

Yes 

HIO 

There  is  a  direct  positive  relationship  between  fear  of  protest  and  the  actual 
PALT. 

Yes 

Hll 

There  is  a  direct  positive  relationship  between  fear  of  protest  and  the  number  of 
Human  Resources  assigned  to  the  source  selection  team. 

No 

H12 

There  is  a  direct  positive  relationship  between  fear  of  protest  and  transaction 
costs. 

Yes 

H13 

The  higher  the  level  of  fear  of  protest,  the  greater  the  number  of  awarded 
contracts. 

No 

H14 

There  is  a  direct  negative  relationship  between  fear  of  protest  and  the  contracting 
officer’s  Authority. 

Yes 
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V.  DISCUSSION  AND  RECOMMENDATIONS 


A.  INTRODUCTION 

The  objective  of  this  research  was  to  address  gaps  in  the  literature  review  and 
offer  acquisition  leaders,  practitioners,  and  scholars  a  better  understanding  of  the  causes 
and  effects  of  a  fear  of  protest.  From  this  research,  acquisition  leaders  and  practitioners 
can  better  align  their  intemal/extemal  policies,  acquisition  strategies,  and  resources  to 
mitigate  the  causes  and  effects  of  fear  of  protest,  which  ultimately  saves  money  and 
increases  customer  satisfaction  (mission)  and  contractor  performance. 

The  existing  literature  (Manuel  &  Schwartz,  2011;  Schwartz  et  ah,  2013; 
O’Rourke,  2014),  Gordon’s  publication  (2013),  and  other  publications  (Knauth,  2013; 
Maser  et  ah,  2010;  Kendall,  2012),  were  used  to  identify  problems  that  have  not  been 
addressed,  specifically,  the  magnitude  of  fear  of  protests,  the  extent  that  fear  of  a  protest 
affects  an  acquisition  strategy,  and  the  lengths  that  acquisition  professionals  would  go  to 
avoid  a  protest.  These  problems,  to  this  point,  have  been  largely  anecdotal.  This  study  is 
the  first  to  assess  the  magnitude  of  fear  of  protest,  and  the  first  to  empirically  explore  its 
antecedents  and  consequence. 

To  close  the  gaps  in  the  existing  literature,  this  research  addressed  the  following 
three  research  questions: 

Research  Question  1:  Are  contracting  ojficers  sujficiently  concerned  about  bid 
protests  to  alter  acquisition  strategies  ? 

Research  Question  2:  If  so,  what  factors  affect  protest  ‘fear”? 

Research  Question  3:  What  are  the  consequences  to  the  acquisition  strategy  and 
contract  outcomes? 

Because  of  the  nature  of  the  research  questions,  it  was  appropriate  to  use  a 
combination  of  qualitative  and  quantitative  methodologies.  Literature  from  several 
governmental,  public,  and  private  studies  and  reports  were  used  to  develop  the  conceptual 
model,  14  hypotheses,  an  interview  questionnaire,  and  an  online  115-question  survey. 
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Once  the  data  was  collected  and  analyzed  from  350  responses  that  represented  a  diverse 
population  of  U.S.  Navy  contracting  officers,  regression  analysis  was  performed  using 
SPSS  Statistics  Version  22.0. 

From  the  statistical  analysis,  many  of  the  hypotheses  had  significant  relationships. 
Overall,  the  magnitude  of  those  relationships  was  relatively  small.  Among  the  more 
significant  outcomes,  the  research  determined  that  fear  of  protest  decreases  the  technical 
evaluation  effectiveness,  which  decreases  the  contractor’s  performance,  and  ultimately 
decreases  buyer  satisfaction.  Additionally,  fear  of  a  protest  is  linked  to  inappropriate  use 
of  the  LPTA  source  selection  method,  to  delays  in  contract  awards,  to  increased 
transaction  costs,  and  to  dissatisfaction  with  discussions  with  offerors. 

B.  DISCUSSION  AND  IMPLICATIONS 

In  this  section,  the  research  results  as  well  as  managerial  implications  are 
discussed.  Additionally,  the  results  of  the  hypothesis  testing  and  regression  analysis  are 
discussed.  The  analysis  resulted  in  statistically  significant  estimates  for  10  of  the 
hypotheses  (Hypotheses  1-10  and  12),  marginally  significant  estimates  for  one  of  the 
hypotheses  (14),  and  estimates  with  no  statistical  support  for  two  of  the  hypotheses  (11 
and  13) 

This  section  is  organized  as  follows.  First,  the  question  of  what  is  the  magnitude 
of  fear  of  protest  among  U.S.  Navy  contracting  officers  was  answered  and  discussed. 
Secondly,  each  result  of  hypothesis  testing  is  discussed  and  explained. 

I.  MagnitudeofFear 

A  histogram  of  the  values  for  fear  of  protest  can  be  found  below  in  Figure  5.  The 
left  side  indicates  less  fear  and  the  right  indicates  more.  For  clarification,  16  respondents 
indicated  that  they  strongly  disagreed,  indicating  a  low  level  of  fear,  and  eight  indicated 
that  they  strongly  agreed,  indicating  a  high  level  of  fear.  The  combined  average  response 
that  indicates  neither  agree  nor  disagree  is  19.71  on  a  possible  scale  range  from  five  to 
35.  This  would  indicate  that,  on  average,  the  respondents  fell  between  neither  disagree  or 
agree.  There  are,  however,  many  respondents  that  fear  a  protest,  and  this  can  have  an 
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effect  on  acquisition  strategies.  Notably,  there  is  a  small  but  distinctly  separate  contingent 
of  respondents  (16,  or  4.5  percent)  who  did  not  fear  a  protest  at  all.  Consistent  with  the 
qualitative  comments  received  in  the  comments  field  of  the  survey  and  in  qualitative 
interviews,  these  strongly-held  perspectives  may  be  personal  philosophies  that  protests 
are  a  natural  part  of  the  acquisition  process.  In  other  words,  why  fear  what  is  expected? 
Importantly,  this  small  contingent  slightly  reduced  the  average  fear,  meaning  that  the 
average  of  the  remainder  of  the  sample  would  be  slightly  higher  at  20.41.  Nonetheless, 
there  appears  not  to  be  a  strong,  pervasive  problem  of  a  fear  of  protest  within  the  Navy. 


Figure  5.  FEAR  Construct  Histogram 

A  low  level  of  a  fear  of  protest  could  be  explained  by  a  contracting  officer’s 
confidence  resulting  from  their  experience  or  in  their  ability  to  properly  conduct  a  source 
selection.  As  one  of  the  interview  participants  pointed  out, 

I  am  not  fearful  of  a  protest  because  if  you  protest  me  and  I  did  it  right,  what — the 
protest  is  not  going  to  be  sustained,  it  is  going  to  be  denied.  So  the  fear  of  a  protest  is 
usually  not  present  other  than  the  delay  in  the  procurement  action  in  case  it  is  an  urgent 
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action  or  something  that  needs  to  get  in  place  and  time.  It  may  delay  the  action,  there  may 
be  a  stay  of  some  sort,  but  other  than  that  avoiding  a  protest — just  doing  the  right  thing 
avoids  a  protest. 

Conventional  wisdom  suggests  that  high  levels  of  exposure  to  something  can 
lower  the  amount  of  fear  one  has  in  it  occurring.  Out  of  an  average  of  13.12  years  of 
contracting  experience,  contracting  officers  only  experienced  an  average  of  1.96  bid 
protests. 

2.  Insufficient  PALT  Planned 

Hypothesis  1:  Insufficient  PALT  has  a  direct  positive  relationship  on  the  level  of 

fear  of  protest. 

There  is  strong  evidence  that  suggests  insufficient  PALT  is  a  contributor  to  the 
level  of  fear  of  protest.  Although  the  relationship  is  significant,  the  coefficient  is  low.  The 
more  insufficient  the  planned  procurement  lead-time  is  thought  to  be,  the  level  of  fear  of 
a  protest  increases.  Alternatively,  when  insufficient  PALT  decreases,  the  level  of  fear 
decreases. 

The  relationship  is  very  plausible.  When  acquisition  personnel  have  less  planned 
PALT  than  they  believe  is  necessary  to  properly  conduct  the  source  selection,  there  is 
less  time  to  adequately  follow  an  acquisition  strategy.  This  insufficient  time  increases  the 
concern  of  a  protest  because  internal  customers  and  acquisition  personnel  may 
unnecessarily  rush  the  acquisition  process.  This  relationship  is  particularly  plausible 
when  the  EOFY  is  approaching.  Customers  must  obligate  their  remaining  funds  before 
the  EOFY,  September  30,  or  lose  the  remaining  funds.  In  some  instances,  customers 
submit  to  the  contracting  agencies  their  requirements  a  few  days  before  the  EOFY.  There 
is  little  time  to  adequately  complete  the  acquisition  process. 

3.  Contracting  Officer’s  Competence 

Hypothesis  2:  The  greater  a  contracting  officer ’s  competence  level,  the  lower  the 

level  of  fear  of  protest. 

In  the  examination  of  a  contracting  officer’s  years  of  experience  as  an  indicator  of 
his  or  her  competence,  there  was  no  relationship  with  a  fear  of  protest.  There  is  strong 
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evidence,  however,  that  a  contracting  officer’s  competence  in  terms  of  the  number  of 
source  selections  experienced  lowers  the  level  of  fear  of  bid  protest.  The  coefficient  path 
of  -.267  suggests  that  there  is  a  negative  relationship  between  the  contracting  officer’s 
source  selection  participation  and  the  level  of  a  fear  of  protest. 

It  was  hypothesized  that  there  would  be  a  relationship  with  both  the  years  of 
experience  and  the  number  of  source  selections  experienced.  However,  the  lack  of  a 
relationship  with  regard  to  years  of  experience  can  be  explained  because  of  a  few  likely 
scenarios.  Experience  in  a  FAR  Part  15  source  selection  can  vary  greatly  regardless  of 
one’s  years  of  experience.  Some  may  have  been  in  the  contracting  career  field  for  many 
years  but  have  rarely  dealt  with  a  FAR  Part  15  source  selection  because  they  have 
worked  mainly  post-award  contract  administration,  pre-award  simplified  acquisitions,  or 
are  in  a  mission  area  whose  requirements  are  mostly  sole  source  procurements  such  as 
research  and  development.  The  number  of  source  selections  is  a  better  determinant  for 
measuring  competence  than  is  years  of  general  contracting  experience.  The  research 
supports  that  the  greater  the  quantity  of  FAR  Part  15  source  selections  a  contracting 
officer  has  experienced,  the  more  comfortable  they  become  in  their  ability  to  work 
through  the  proper  process  and  ultimately  have  less  fear  of  a  bid  protest. 

The  empirical  data  also  did  not  support  the  DAWIA  certification  level  as  a 
surrogate  measure  of  competence.  No  relationship  between  DAWIA  certification  level 
and  fear  of  protest  was  found.  There  can  very  well  be  a  contracting  officer  with  a  level 
two  DAWIA  certification  that  has  more  source  selection  experience  than  a  level  three 
DAWIA  certified  contracting  officer.  DAWIA  level  is  not  a  reliable  indicator  of 
experience,  capability,  and  therefore,  a  KO’s  confidence  in  ability. 

It  is  important  for  practitioners  to  understand  that  competence  comes  in  many 
forms  and  the  person  selected  for  a  source  selection  of  importance  should  not  be  chosen 
on  DIAWA  level  or  years  of  general  contracting  experience  alone;  but  they  should  also 
look  at  how  many  source  selections  the  contracting  officer  has  experienced. 
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4.  Requirement  Criticality/Importance 

Hypothesis  3:  The  level  of  requirement  criticality/importance  has  a  positive 

impact  on  the  level  of  fear  of  protest. 

There  is  overwhelming  evidence  that  requirement  criticality/importance  is  a 
contributor  to  the  level  of  fear.  The  relationship  for  Hypothesis  3  is  statistically 
significant  and  the  path  coefficient  is  positive  and  fairly  strong,  Therefore,  as  the  level  of 
the  requirements  critically /importance  increases,  the  level  of  fear  of  protest  increases. 
There  is  much  more  concern  and  pressure  to  ensure  the  customer  gets  the  requested 
requirement  in  a  timely  manner  when  the  requirement  is  critically  important  to  their 
mission.  Protests  take  time  and  delay  the  customer’s  receipt  of  his  or  her  required 
products  or  services.  If  the  requirement  is  of  great  importance  to  the  customer,  the 
pressure  from  this  importance  is  felt  by  the  contracting  officer.  This  can  increase  the 
contracting  officer’s  level  of  a  fear  of  protest.  Managers  should  be  aware  of  this  and 
ensure  that  proper  planned  PALT  is  built  into  the  acquisition  strategy  to  counter  the 
increase  of  a  fear  of  protest  and  to  ensure  enough  time  for  proper  peer  reviews. 

5.  Protest  Risk 

Hypothesis  4:  As  protest  risk  increases,  the  level  of  fear  of  protest  increases. 

There  is  strong  evidence  suggesting  that  protest  risk  is  a  contributor  to  the  level  of 
fear  of  protest.  The  relationship  for  Hypothesis  4  is  statistically  significant,  and  the  beta 
coefficient  is  positive.  As  the  level  of  undesirable  consequences  increases,  the  level  of 
fear  increases.  There  is  a  perception  that  a  bid  protest  reflects  poorly  on  the  contracting 
officer,  as  demonstrated  by  the  following  quote  by  one  of  the  contracting  officers 
interviewed:  ‘“oh,  you  got  a  protest.  What  did  you  do  wrong?’  kind  of  thing.  I  don’t  think 
there  is  validity  to  that,  but  I  think  that  sometimes  it  is  looked  at,  oh  so  you  got  a  protest 
so  maybe  they  didn’t  do  something  right  in  there.”  Another  contracting  officer 
interviewed  expressed  that,  “Well,  first  off  is  that  it  reflects  poorly  on  the  command.  It 
gives  the  command  a  bad  reputation  in  terms  of  being  able  to  follow  the  source  selection 
plan,  for  example,  professionalism  of  the  contract  specialist  and  contracting  officers  and 
the  source  selection  team  and  all  of  the  people  that  contribute  to  the  award  decision. 
Secondly,  protests  they  take  a  lot  of  time.” 
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On  the  other  hand,  in  some  commands  it  is  considered  a  career  milestone  to  have 
experience  with  a  protest  for  promotion  to  GS-13,  as  was  expressed  in  our  interviews. 
Another  common  theme  throughout  our  interviews,  and  as  indicated  in  our  survey 
responses,  is  the  increased  workload  a  protest  demands.  Everyone  is  busy,  and  the  last 
thing  needed  is  additional  work.  Furthermore,  new  requirements  do  not  stop  coming  just 
because  a  protest  occurred;  a  protest  can  significantly  increase  an  already  overburdened 
work  load. 

There  are  several  consequences  to  the  acquisition  strategy  and  contract  outcomes 
as  a  result  of  fear.  The  following  hypotheses  focus  on  the  results  relating  to  these 
consequences. 


6.  Technical  Evaluation  Effectiveness 

Hypothesis  5:  The  level  of  fear  of  protest  has  a  negative  effect  on  the  technical 
evaluations  effectiveness. 

As  a  result  of  fear,  there  is  statistical  evidence  that  supports  a  negative  effect  on 
the  effectiveness  of  technical  evaluations.  The  negative  relationship  between  fear  of 
protest  and  effectiveness  of  technical  evaluations  is  small,  however.  The  relationship 
suggests  that  as  a  person’s  level  of  a  fear  of  protest  increased,  the  effectiveness  of  the 
technical  evaluation  decreased.  From  the  research  data,  on  a  scale  of  one  to  seven  with 
one  being  strongly  disagree  and  seven  being  strongly  agree,  responses  averaged  4.5  on 
this  scale  that  measured  “At  least  one  technical  evaluator  expressed  concern  about  not 
being  able  to  say  what  needs  to  be  said  in  the  technical  evaluation.”  contracting  officers 
somewhat  agreed  with  an  average  of  4.63  on  the  scale  that  “At  least  one  technical 
evaluator  was  concerned  that  the  constraints  imposed  on  their  evaluations  impeded 
his/her  ability  to  write  a  meaningful  evaluation.”  This  suggests  that  the  technical 
evaluators  do  not  believe  the  process  is  sufficiently  optimized.  Additionally,  on  the  scale 
for  “Upon  evaluation  of  proposals,  at  least  one  technical  evaluator  expressed  a  need  to 
change  at  least  one  evaluation  criterion  or  its  definition.”  The  contracting  officer 
somewhat  agreed  with  this  with  an  average  of  4.53.  It  is  plausible  that  a  contracting 
officer’s  fear  of  protest  could  lead  them  to  change  the  wording  or  the  standards  of  the 
technical  evaluations.  To  a  certain  degree,  without  the  threat  of  a  bid  protest,  the  quality 
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of  the  technical  evaluation  would  change.  The  statistical  analysis  suggests  that  the 
effectiveness  of  the  technical  evaluations  decreases  as  the  level  of  fear  increases. 

7.  Source  Selection  Method  Fit/Appropriateness 

Hypothesis  6:  There  is  a  direct  positive  relationship  between  fear  of  protest  and 

the  Source  Selection  method  fit/appropriateness. 

Hypothesis  6  was  partially  supported  and  tests  were  consistent  with  the 
relationship  hypothesized.  The  hypothesis  measured  fear  of  protests  effect  on  both 
perceived  Source  Selection  Method  Appropriateness  and  on  multiple  aspects  of  the 
source  selection  strategy.  The  relationship  between  fear  of  protests  and  source  selection 
method  appropriateness  was  not  supported.  The  appropriateness  construct  consisted  of 
questions  relating  to  the  acquisition  strategy  used  on  the  particular  source  selection. 
Acquisition  strategy  is  a  broad  topic  that  encompasses  decisions  made  by  various 
stakeholders  early  on  in  the  process  of  fulfilling  a  requirement.  It  is  plausible  that  these 
decisions  are  made  so  early  that  the  factors  that  contribute  to  fear  of  protest  have  not  been 
solidified. 

Several  relationships  between  fear  of  protest  and  various  aspects  of  the  source 
selection  strategy  were  found,  however  perhaps  the  most  telling  evidence  of  the 
deleterious  effects  that  fear  of  protest  was  its  impact  on  LPTA  inappropriateness. 
Significant  findings  included 

•  Method  inappropriateness — While  it  represents  only  7.5  percent  of  the 
respondents  that  used  either  LPTA  or  trade-off  methods,  it  must  be  noted 
that  23  respondents  used  a  source  selection  method  that  they  felt  was  to 
some  degree  inappropriate.  The  relationship  between  fear  of  protests  and 
the  inappropriate  use  of  LPTA  was  significant.  The  results  show  that 
practitioners  with  high  fear  of  protests  are  four  times  more  likely  to 
inappropriately  use  LPTA. 

•  Negative  effects  on  satisfaction  with  discussions — significant  relationships 
were  found  between  fear  of  protests  and  satisfaction  with  discussions. 
When  fear  of  protests  exists,  the  level  of  satisfaction  to  freely  and  openly 
conduct  discussions  with  an  offeror  diminishes.  While  the  effects  were 
marginal,  the  mere  existence  of  this  relationship  is  significant.  When 
respondents  were  measured  for  the  extent  that  they  would  change 
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discussions  if  there  were  no  ability  to  protest,  the  results  were  also 
significant.  The  results  indicated  there  would  be  a  distinct  change  in 
discussions  if  the  protest  system  did  not  exist. 

The  negative  effects  that  the  fear  of  protest  has  on  source  selection  decision¬ 
making  can  be  seen  most  in  the  variables  measured  to  test  this  hypothesis.  Contracting 
professionals  are  trained  and  trusted  to  make  decisions  that  support  the  best  interest  of  the 
government.  It  was  statistically  shown  that  fear  of  protests  does  have  a  negative  effect  on 
acquisition  strategy  decisions.  However,  it  must  be  noted  that  the  magnitudes  of  effects  is 
not  substantial. 


8.  Contractor  Performance 

Hypothesis  7:  There  is  a  positive  relationship  between  the  fit/appropriateness  of  a 
source  selection  and  contractor  performance. 

Hypothesis  8:  There  is  a  positive  relationship  between  technical  evaluation 
effectiveness  and  contractor  performance. 

Although  source  selection  method  appropriateness  was  not  statistically  significant 
in  its  relationship  with  fear  of  a  protest.  Hypothesis  7  was  significant.  There  is  a  positive 
relationship  between  perceived  source  selection  method  appropriateness  and  contractor 
performance.  As  acquisition  strategies  are  properly  tailored  to  fit  requirements,  contractor 
performance  tends  to  increase  as  a  result.  In  assessing  source  selection  method 
appropriateness,  survey  respondents  were  prompted  to  consider  the  buying  situation  (e.g., 
complexity,  dollar  value,  acquisition  objectives,  contract  length,  performance  risk, 
criticality  to  the  mission,  availability  of  supply,  time  available  to  award  a  contract,  etc.). 
They  were  also  asked  to  rate  how  well  the  strategy  would  achieve  the  acquisition 
objectives  and  how  well  the  strategy  would  help  select  the  best  offeror. 

Hypothesis  8  was  also  statistically  significant.  As  the  effectiveness  of  technical 
evaluations  increases,  so  does  the  contractor  performance.  Hence,  as  more  effective 
technical  evaluations  are  generated,  the  more  apt  the  source  selection  team  is  to  select  the 
contractor(s)  that  can  perform  the  work  well.  The  contractor  that  has  the  best  value  to  the 
government  is  selected  for  the  requirement,  which  will  ultimately  lead  to  greater 
contractor  performance. 
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9.  Buyer  Satisfaction 

Hypothesis  9:  There  is  a  direct  positive  relationship  between  contractor’s 
performance  based  on  the  source  selection  method  used  and  the  Buyer’s 
satisfaction. 

There  is  overwhelming  evidence  that  suggest  that  the  contractor’s  performance 
contributes  to  an  increase  in  the  buyer’s  satisfaction.  The  relationship  suggests  that,  as  a 
contractor’s  performance  increases,  buyer’s  satisfaction  increases.  The  contracting 
officer’s  responsiveness  and  willingness  to  address  the  concerns  of  the  customer  can  lead 
to  a  better  suited  acquisition  strategy  and  have  a  positive  effect  on  customer  satisfaction. 
The  contracting  officer  is  working  on  behalf  of  the  customer  and  the  customer  is  the 
lifeblood  of  the  contracting  activity.  It  is  in  the  best  interest  of  the  contracting  activity  to 
ensure  proper  contract  performance  by  using  the  appropriate  contract  vehicle,  source 
selection  method,  and  overall  acquisition  strategy.  This  will,  in  turn,  improve  customer 
satisfaction  and  increase  their  public  value  to  the  taxpayer.  Therefore,  it  is  imperative  for 
contracting  officers  to  select  the  appropriate  source  selection  method  to  choose  the  best 
Contractor  to  satisfy  the  buyer’s  requirements  and  expectations. 

10.  PALT  Actual 

Hypothesis  10:  There  is  a  direct  positive  relationship  between  fear  of  protest  and 
the  actual  PALT. 

There  is  statistical  evidence  that  suggests  that  a  fear  of  protest  contributes  to  an 
increase  in  actual  PALT.  The  data  suggests  that  there  is  a  slightly  greater  than  50  percent 
chance  that  the  actual  PALT  will  increase  as  a  result  of  fear  of  protest.  If  contracting 
officers  are  concerned  about  a  protest,  then  they  will  spend  more  time  to  justify  their 
decisions.  They  could  seek  assistance  from  other  acquisition  personnel  for  their  opinion 
and  recommendations,  which  can  increase  actual  PALT.  They  can  also  refer  to  their  legal 
advisors  and  supervisors  who  likely  recommend  additional  steps  or  reworked  documents 
that,  in  turn,  ultimately  increasing  actual  PALT.  From  the  data,  the  average  PALT 
planned  was  182.97  days  (6.09  months).  The  average  actual  PALT  was  236.72  days  (7.89 
months).  The  difference  is  53.75  days  (1.79  months).  Added  time  is  added  money  (i.e., 
transaction  costs).  Thus,  efficiency  is  compromised  with  greater  fear  of  protest.  The  data 
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suggests  that  some  of  this  additional  time  can  be  attributed  to  a  fear  of  protest.  Although 
not  tested,  this  can  have  an  effect  of  customer  satisfaction  and  delay  mission  capabilities. 

11.  Resources 

Hypothesis  11:  There  is  a  direct  positive  relationship  between  fear  of  protest  and 

the  number  of  Human  Resources  assigned  to  the  source  selection  team. 

There  is  statistical  evidence  that  supports  a  direct  positive  relationship  between 
protest  fear  and  the  number  of  personnel  assigned  to  the  source  selection  team  (boards, 
contracting  team,  price  analysis  team,  past  performance  team,  legal  team,  and  technical 
evaluation  team).  Additionally,  the  research  data  captures  an  average  of  8.96  people  are 
on  the  source  selection  team.  Without  the  risk  of  protest,  contracting  officers  believe  that 
an  average  of  7.65  people  should  have  been  on  the  source  selection  team.  The  difference 
is  1.33  people.  Additionally,  110  survey  respondents  indicated  that  they  had  more  people 
than  what  they  believed  was  needed.  More  people  translate  into  more  transaction  costs. 
The  results  suggest  that  at  least  one  person  too  many  is  assigned.  Perhaps  more  attention 
is  needed  to  not  over-staff  teams  thereby  avoid  wasted  resources.  Additionally,  if  fear  of 
protest  can  be  reduced,  team  size  can  be  reduced  and  transaction  costs  can  be  avoided. 
While  these  salary  costs  may  be  dismissed  as  sunk  costs,  certainly  excess  personnel  could 
accomplish  other  pertinent  work  if  not  serving  on  the  source  selection  team. 

12.  Transaction  Costs 

Hypothesis  12:  There  is  a  direct  positive  relationship  between  fear  of  protest  and 

transaction  costs. 

There  is  strong  evidence  that  a  fear  of  protest  causes  a  small  increase  in 
transaction  cost.  The  average  cost  per  transaction  was  $235,236  with  a  standard  deviation 
of  $291,620.  The  research  also  uncovered  that  for  every  dollar  awarded  in  a  source 
selection  there  is  an  average  of  0.08  cents  in  personnel  cost.  Without  a  known 
benchmark,  it  is  difficult  to  generalize  the  magnitude  of  these  costs. 

As  the  level  of  fear  of  a  protest  increase,  the  cost  devoted  to  the  source  selection 
process  increases  as  well.  Furthermore,  as  the  fear  of  a  protest  increases,  so  do  the 
iterations  of  common  source  selection  documents.  The  data  showed  that  a  fear  of  protest 
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directly  caused  a  small  increase  in  the  amount  of  iterations  of  the  following  documents: 
source  selection  plan,  debriefing  scripts,  color  and  rating  charts,  and  evaluation 
debriefing  charts.  These  documents  must  be  completed  and  reviewed  by  human  capital 
costing  additional  time  and  money  devoted  to  a  source  selection.  We  found  very  little 
qualitative  evidence  that  transaction  costs  were  of  any  concern  to  interview  informants. 
The  maxim  appears  to  be  to  defend  the  acquisition  against  a  protest  no  matter  the  cost  in 
terms  of  time  and  effort. 

13.  Number  of  Awarded  Contracts 

Hypothesis  13:  The  higher  the  level  of  fear  of  protest,  the  greater  the  number  of 

awarded  contracts. 

There  is  no  statistical  evidence  to  suggest  that  the  higher  the  level  of  fear,  the 
greater  the  number  of  awarded  contracts.  From  the  data,  contracting  officers  are  awarding 
about  the  same  amount  of  contracts  that  they  planned.  An  average  of  2.24  contracts  was 
planned.  An  average  of  2.39  contracts  was  awarded.  There  is  a  small  difference  of  0.15, 
but  our  data  suggests  that  fear  is  not  a  factor  in  the  number  of  awarded  contracts.  As 
reported  in  the  trade  literature,  there  are  instances  of  awarding  more  contracts  in  order  to 
thwart  a  specific  protest,  but  this  practice  appears  not  to  be  pervasive. 

14.  Contracting  Officer  Authority 

Hypothesis  14:  There  is  a  direct  negative  relationship  between  fear  of  protest  and 

the  contracting  officer ’s  authority. 

Although  the  statistical  evidence  is  marginally  significant,  there  is  some  evidence 
that  fear  of  protest  diminishes  contracting  officers’  authority  (i.e.,  discretion  in  making 
decisions). 

This  relationship  has  plausibility.  As  contracting  officers  are  more  concerned 
about  a  protest,  they  seek  assistance  in  avoiding  a  protest,  which  could  diminish  their 
discretion  in  making  decisions.  If  a  contracting  officer  goes  to  the  legal  advisor.  Legal 
could  influence  the  contracting  officer’s  decision  through  their  legal  opinion  and 
recommendations.  Often,  Legal  and  committee  advisors  will  be  conservative  and 
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recommend  wording  changes  to  documents,  changes  to  ratings,  amendments  to  the  RFP 
and  subsequent  invitations  for  revised  proposals,  and/or  further  discussions  to  clear  up 
any  uncertainty  in  evaluations. 

Additionally,  if  a  contracting  officer  has  a  high  level  of  fear  of  protest,  other 
acquisition  team  members  likely  do  as  well.  A  high  concern  of  a  protest  could  cause 
others  not  to  empower,  trust,  or  support  the  contracting  officer  with  making  the  required 
decisions,  which  could  diminish  a  Contracting  officer’s  discretion  in  making  decisions. 
During  the  interviews,  it  appeared  that  the  level  of  which  the  contracting  officer  felt 
compelled  to  take  the  advice  of  legal  as  “gospel”  depended  on  the  contracting  officer. 
Some  felt  as  though  they  were  compelled  to  take  the  advice  of  legal  and  some  felt  it  was 
within  their  discretion  to  do  what  they,  as  the  contracting,  thought  was  the  best  course  of 
action. 

It  is  ultimately  obviously  the  contracting  officer’s  decision  and  it  basically 
depends  on  what  their  comfort  level  is  in  terms  of  some  won’t  turn  their  head  without 
getting  an  opinion  from  legal  and  making  sure  that  they  sign  off  on  it  and  other  ones 
basically  believe — and  I  agree — that  legal  is  an  advisory  role,  obviously. 

It  has  been  noted  during  our  interviews  that  the  legal  department  is  the  most 
concerned  about  a  protest.  This  makes  sense  since  they  will  carry  a  lot  of  the  work 
involved  in  a  protest  if  received.  It  has  also  been  stated  in  our  interviews  that  legal 
advocates  for  a  LPTA  source  selection  methodology.  Since  our  data  suggests  that  there  is 
a  very  weak  negative  relationship  with  a  fear  of  protest  and  contracting  officer’s 
authority,  we  can  infer  that  the  contracting  officers  are  not  unnecessarily  influenced  by 
legal,  but  there  were  13  cases  in  our  350  respondents  in  which  the  contracting  officer 
chose  LPTA  even  though  they  believed  it  was  not  the  best  course  of  action. 

C.  ANSWERS  TO  RESEARCH  QUESTIONS 

1.  Research  Question  1:  Are  Contracting  Officers  Sufficiently 
Concerned  about  Bid  Protests  to  Alter  Acquisition  Strategies? 

The  answer  to  this  research  question  is  “ycj'.”  U.S.  Navy  contracting  officers 
change  aspects  of  acquisition  strategies  to  avoid  a  protest.  However,  U.S.  Navy 
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contracting  officers  appear  not  to  be  overly  concerned  about  a  bid  protests.  The  concerns 
of  protest  exist,  but  overall  the  magnitude  is  not  excessive.  There  is  little  evidence  from 
the  research  suggesting  that  contracting  officers  drastically  alter  acquisition  strategies  to 
avoid  a  protest. 

There  is  evidence  to  suggest  that  a  fear  of  protest  can  alter  an  acquisition  strategy. 
The  350  survey  respondents,  who  have  an  average  of  13.12  years  of  contracting 
experience,  were  asked  throughout  their  career,  how  many  times  they  have  awarded  a 
task/delivery  order  against  an  IDIQ  contract  (or  Blanket  Purchase  Agreement  [BPA])  in 
order  to  avoid  a  bid  protest.  The  data  shows  that  88  respondents  had  done  so  throughout 
their  career  with  4,139  contracting  actions.  It  was  also  shown  that  67  respondents  avoided 
a  protest  using  a  sole  source  award  throughout  their  career  in  order  to  (1011  contracting 
actions)  and  80  respondents  had  modified  an  existing  contract  throughout  their  career  in 
order  to  avoid  a  protestable  competitive  procurement  (1,065  contracting  action).  With 
350  respondents  that  have  an  average  of  13.12  years  of  experience,  this  equals  4,784.5 
years  of  total  experience.  The  total  contracting  actions  from  the  three  previously 
discussed  survey  questions  totals  6,215.  This  is  just  under  1.3  redirected  contracting 
actions  per  year  of  experience.  There  are  consequences  of  a  fear  of  protest  that  our 
research  revealed.  These  consequences  are  discussed  in  research  question  three. 

2.  Research  Question  2:  If  So,  What  Factors  Affect  Protest  “Fear”? 

Based  on  the  regression  model,  insufficient  PALT,  protest  risk,  the  number  of 
source  selections  a  contracting  officer  previously  experienced,  and  the  requirement 
criticality/importance  are  the  factors  that  affect  a  fear  of  protest.  Neither  a  Contracting 
officer’s  years  of  contracting  experience  nor  their  DAWIA  certification  level  affected  the 
fear  of  protest. 


3.  Research  Question  3:  What  are  the  Consequences  to  the  Acquisition 
Strategy  and  Contract  Outcomes? 

Based  on  the  significant  levels,  a  fear  of  protest  can  have  negative  consequences 
on  the  effectiveness  of  technical  evaluations  leading  to  a  decrease  in  contractor 
performance  and,  in  turn,  buyer  satisfaction.  An  increase  of  fear  of  protest  has  a  negative 
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effect  on  technical  evaluations  effectiveness.  Technical  evaluation  effectiveness  has  a 
positive  influence  on  contractor  performance  which,  in  turn,  is  positively  related  to  buyer 
satisfaction.  Technical  evaluation  effectiveness  considered  elements  such  as  (a)  being 
able  to  say  what  needs  to  be  said,  (b)  being  able  to  write  a  meaningful  evaluation,  and  (c) 
a  need  to  change  an  evaluation  factor  or  its  definition. 

The  perceived  appropriateness  of  the  source  selection  method  will  also  have  a 
positive  impact  on  contractor  performance.  The  more  appropriate  the  source  selection 
method  used,  the  higher  the  contractor  performance  leading  to  greater  buyer  satisfaction. 
A  notable  finding  from  this  research  is  that  the  inappropriate  use  of  LPTA  was  shown  to 
have  a  substantial  negative  impact  on  contractor  performance. 

Actual  PALT  can  increase  as  a  result  of  a  fear  of  protest.  The  research  data 
revealed  that  there  is  a  50  percent  chance  that  PALT  will  increase  as  a  result  of  fear. 
Thus,  efficiency  is  compromised  with  greater  fear  of  protest.  The  data  suggests  that  some 
of  this  additional  time  can  be  attributed  to  a  fear  of  protest.  Although  not  tested,  this  can 
have  an  effect  of  customer  satisfaction  and  delay  mission  capabilities. 

Contracting  officers  perceive  their  authority  to  decrease  as  their  fear  of  protest 
increases.  A  plausible  explanation  can  be  that  as  contracting  officers  are  more  concerned 
about  a  protest;  they  seek  assistance  in  avoiding  a  protest,  which  could  diminish  their 
discretion  in  making  decisions.  Additionally,  a  high  concern  of  a  protest  could  cause 
others  not  to  empower,  trust,  or  support  the  contracting  officer  with  making  the  required 
decisions,  which  could  diminish  a  contracting  officer’s  discretion  in  making  decisions. 

Fear  of  protest  also  has  a  negative  impact  on  the  level  of  satisfaction  with 
discussions.  As  fear  increases,  the  contracting  officer’s  satisfaction  with  discussions 
decreases.  This  is  an  indicator  of  suboptimal  discussion  that  could  have  implications  for 
the  awarded  contract  and,  more  importantly,  contractor  performance.  Another 
consequence  of  a  fear  of  protest  is  the  transaction  cost  and  time  associated  with  the 
source  selection.  As  a  fear  of  protest  increases,  so  do  the  iterations  of  some  documents. 
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such  as  source  selection  plan,  debriefing  scripts,  color  and  rating  charts,  and  evaluation 
debriefing  charts.  This  can  lead  to  additional  time  and  money  (i.e.,  opportunity  costs) 
spent  on  the  requirement. 

D.  IMPLICATIONS  AND  RECOMMENDATIONS  FOR  PRACTITIONERS 

This  section  discusses  some  of  the  possible  implications  this  study  revealed  and 
what  managers  and  practitioners  can  learn  from  them.  As  discussed  earlier,  the  magnitude 
of  fear  of  protest  does  not  appear  to  be  a  pervasive  problem  within  the  Navy’s  contracting 
community,  but  there  are  many  contracting  officers  that  allow  this  fear  to  affect  elements 
of  their  acquisition  strategy.  There  were  23  survey  respondents  (7.49  percent)  that 
revealed  that  the  source  selection  method  used  was  to  some  degree  inappropriate.  While 
this  proportion  appears  insignificant,  it  can  be  argued  that  any  single  instance  of  an 
inappropriate  source  selection  method  gives  room  for  pause  to  see  what  is  driving  this 
inappropriateness.  As  mentioned  by  an  interview  participant,  “I  will  tell  you,  legal  pushes 
the  LPTA.  They  push  it  a  lot.”  Another  interviewee  mentioned.  At  this  juncture,  there  are 
too  many  hands  in  the  soup,  and  the  Procuring  Contracting  Officer  (PCO)  authority  has 
been  diminished.  Attorneys  need  to  resume  the  role  of  counselors  again.  Fear  of  protests 
does  not  generate  good  contracts,  or  in  this  case  can  prevent  any  award  from  being  made. 

Recommendation  1:  Agencies  should  foster  a  culture  that  empowers  the 

contracting  officer  to  perform  the  job  they  are  trained  to  accomplish. 

Since  the  source  selection  method  is  not  a  matter  of  legal  sufficiency.  Legal 
should  not  determine  the  source  selection  method  used,  nor  should  they  encourage  any 
particular  source  selection  method  such  as  LPTA.  Selecting  the  source  selection  method 
is  a  contracting  officer’s  decision  based  on  experience,  knowledge,  and  professional 
judgment;  contracting  officers  should  feel  empowered  and  should  in  no  way  be 
influenced  against  their  better  judgment.  Otherwise,  Federal  Government  agencies  may 
have  a  professional  field  with  a  high  degree  of  accountability  but  subtly-diminished 
authority.  Acquisition  leaders  should  reassure  their  contracting  officers  that  they  have  the 
ultimate  say  in  the  source  selection  method  employed,  and  that  legal’s  role  is  to  advise 
and  to  clarify  the  many  legal  issues  that  contracting  within  the  government  entails.  If  the 
contracting  officer  does  not  have  the  authority  or  believes  that  the  legal  department  has 
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the  ultimate  say,  it  would  be  less  expensive  to  the  government  to  hire  clerks  or  paralegals 
to  accomplish  the  contracting  mission.  The  judgment  of  the  contracting  officer,  the 
person  actually  signing  the  contract,  should  be  respected  by  management  and 
management  should  encourage  a  culture  that  empowers  the  contracting  officer. 

Undue  influence  from  management  can  also  lead  to  an  inappropriate  source 
selection  method.  There  is  pressure  applied  to  management  to  reduce  their  unit’s  average 
time  for  procurements.  This  can  lead  to  pressure  from  management  to  the  contracting 
officer  resulting  in  the  overuse  of  LPTA.  Management  should  resist  this  pressure.  Our 
data  suggests  that  LPTA  inappropriateness  can  have  a  negative  effect  on  contractor 
performance  that  leads  to  less  buyer  satisfaction.  Ultimately  the  best  course  of  action  is  to 
use  the  source  selection  method  that  best  suits  the  requirement. 

Recommendation  2:  Agencies  should  establish  a  standard  policy  for  PALT  based 

on  dollar  value  and  source  selection  method. 

Planned  PALT  is  often  a  result  of  the  internal  customer’s  situation.  It  may  be  a 
requirement  that  is  needed  within  three  weeks  or  three  months,  but  most  everyone  wants 
their  requirement  fulfilled  as  quickly  as  possible.  It  is  important  that  the  contracting 
officer  has  the  ability  to  manage  the  expectations  of  the  customer  and  that  the  contracting 
officer’s  management  supports  them  on  this  issue.  If  planned  PALT  is  less  than  what  the 
contracting  officer  believes  to  be  sufficient,  there  are  implications  that  a  fear  of  a  protest 
will  increase.  This,  in  turn,  can  ultimately  have  a  negative  effect  on  technical  evaluation 
effectiveness  and  ultimately  on  the  contractor’s  performance.  A  possible  solution  to  this 
is  to  have  an  instruction  that  sets  PALT  by  dollar  value  and  by  source  selection  method. 
A  higher  dollar  requirement  that  is  best  procured  using  a  trade-off  approach  will  take 
more  time  and  should  be  built  in  to  the  planned  PALT.  The  contracting  officer  should 
have  the  flexibility  to  deviate  from  these  predetermined  PALT’s  based  on  their 
experience  and  judgment. 

Recommendation  3:  Agencies  should  establish  a  training  program  to  supplement 

FAR  Part  15  source  selection  experience. 

The  competence  level  of  the  contracting  officer  is  validly  measured  in  terms  of 
the  number  of  source  selections  experienced  (and  not  raw  years  of  contracting  experience 
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or  DAWIA  certification  level).  As  the  number  of  source  selections  conducted  increases, 
the  level  of  fear  of  protest  decreases.  Therefore,  efforts  should  be  made  to  increase  the 
number  of  source  selections  experienced  by  contracting  officers.  Of  the  350  survey 
respondents,  it  was  found  that  the  average  years  of  experience  are  13.12  with  an  average 
of  18.31  source  selection  participations.  That  is  just  under  1.40  source  selections  a  year. 
There  is  no  equal  alternative  to  on-the-job-training  (OJT),  but  source  selection 
simulations  and  scenario-based  training  could  be  utilized  as  an  alternative  and  as  a 
supplement  to  OJT.  If  the  acquisition  community  is  relying  solely  on  OJT,  it  can  take  a 
contracting  officer  far  too  long  to  gain  an  adequate  level  of  competence  with  FAR  Part  15 
source  selections.  In  addition,  when  selecting  a  contracting  officer  for  a  source  selection, 
it  is  important  to  realize  that  years  of  contracting  experience  and  DAWIA  certification 
level  do  not  represent  his  or  her  level  of  competence. 

As  criticality  and  importance  of  the  requirement  increases,  so  does  the  contracting 
officer’s  level  of  a  fear  of  protest.  As  fear  increases,  buyer  satisfaction  can  be  negatively 
altered  through  the  inappropriate  use  of  source  selection  methods  and  the  level  at  which 
they  effectively  communicate  with  the  offerors.  Therefore,  the  criticality  and  importance 
of  the  requirement  should  be  taken  into  account  when  assigning  a  contracting  officer  to 
serve  as  the  PCO  for  the  source  selection.  Perhaps  a  PCO  with  greater  source  selection 
experience  (i.e.,  which  lowers  fear  of  protest)  can  offset  the  higher  fear  of  protest 
associated  with  more  critical  requirements. 

As  a  fear  of  protest  increases,  the  effectiveness  of  technical  evaluations  decreases. 
Recall  that  this  construct  encapsulated  situations  where  technical  evaluators  expressed 
concerns  that  (a)  they  did  not  have  the  latitude  to  say  what  needed  to  be  said  in  their 
evaluations,  (b)  constraints  impeded  the  ability  to  write  a  meaningful  evaluation,  and  (c) 
an  evaluation  factor  or  its  definition  needed  to  be  changed  after  receipt  of  proposals.  This 
should  be  cause  for  alarm  since  the  purpose  of  a  source  selection  is  to  attain  a  needed 
performance  level  outcome  at  a  reasonable  price/cost  while  mitigating  risk.  Technical 
evaluators  via  the  contracting  officer  should  have  the  latitude  and  the  freedom  to  halt  a 
source  selection  to  ensure  the  customer  and  the  tax  payer  are  getting  the  best  contracted 
outcome  (i.e.  value)  for  the  invested  dollars;  though  this  recommendation  is  time 
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consuming  and  is  often  not  an  option.  Evaluators,  however,  can  reduce  the  risk  of  this 
occurring  by  becoming  part  of  the  crafting  of  Section  M  criteria.  Is  a  FAR  Part  15  full 
trade-off  source  selection  too  cumbersome  to  ensure  we  are  getting  the  best  requirement 
outcome?  Is  the  transaction  cost  of  full  trade  off  too  high?  It  may  be  that  the  acquisition 
rules  on  evaluations  and  the  protest  implications  of  them  are  too  strict  and  FAR  part  15 
may  need  another  reevaluation  to  allow  for  more  flexibility.  Our  research  suggests  that  as 
a  contracting  officer’s  fear  of  a  protest  increases,  there  is  a  decrease  in  technical 
evaluation  effectiveness  leading  to  a  decrease  in  contractor  performance  and  less  buyer 
satisfaction. 

Recommendation  4:  The  DAU  should  establish  a  prerequisite  online  training 

program  for  technical  evaluation  team  members. 

Additional  training  for  the  technical  evaluators  and  for  the  technical  evaluation 
team  is  required  to  increase  their  level  of  competence  within  the  evaluation  process.  The 
evaluation  process  has  many  people  that  are  not  necessarily  familiar  with  the  rules  that 
govern  this  process,  and  it  is  ultimately  the  responsibility  of  the  contracting  officer  to 
ensure  that  the  technical  team  is  aware  and  its  members  are  sufficiently  trained  to  handle 
the  evaluation  process.  The  contracting  officer  should  not  be  solely  responsible  to  train 
the  technical  evaluators,  and  all  people  involved  in  the  process  should  have  a  minimum 
training  requirement  that  educates  them  on  the  process.  All  members  involved  in  the 
technical  evaluation  process  should  have  a  minimum  uniform  standard  of  training.  A 
DOD  or  a  DAWIA  certification  could  give  the  contracting  officer  the  confidence  that  the 
technical  evaluation  team  has  the  proper  level  of  knowledge  to  carry  out  the  evaluation 
function.  If  the  contracting  officer  is  comfortable  with  the  level  of  knowledge  then  he  or 
she  can  focus  on  guiding  and  advising  the  technical  evaluators  and  spend  less  time 
elevating  their  baseline  knowledge  of  the  process.  This  can  reduce  the  fear  a  contracting 
officer  has  of  a  protest  and  increase  the  effectiveness  of  technical  evaluations.  This  can, 
in  turn,  lead  to  increased  contractor  performance  and  buyer  satisfaction. 

E.  LIMITATIONS  OF  THE  STUDY 

This  study  had  several  limitations.  First,  this  study  was  limited  to  the  Navy’s 
acquisition  workforce.  Thus,  while  the  fear  of  a  protest  phenomenon  has  been  discussed 
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in  the  context  of  all  Federal  contracting,  these  study  results  may  not  be  generalized 
beyond  the  U.S.  Navy.  Respondents  were  asked  to  answer  questions  that  reflected  only 
their  most  recent  completed  FAR  Part  15  source  selection  above  $150,000  in  which  they 
participated.  On  certain  questions,  respondents  provided  estimated  responses.  For 
example,  a  respondent  stated  that  a  GS-15  Supervisor  spend  about  10  percent  of  one 
year’s  time  dedicated  to  the  source  selection.  It  may  be  difficult  to  for  respondents  to 
estimate  accurate  times;  therefore,  these  estimated  responses  may  not  be  precise. 
However,  these  estimates  were  useful  in  establishing  a  benchmark  to  understand  the 
monetary  cost  incurred  to  avoid  a  protest.  Activity -based  costing  using  a  database  (i.e., 
the  Navy’s  Standard  Labor  Data  Collection  and  Distribution  Application(  SLDCADA)) 
to  retrieve  data  that  captured  the  amount  of  time  a  government  employee  worked  on  a 
particular  source  selection  could  reveal  a  more  accurate  monetary  cost  to  avoid  a  protest. 

Additionally,  contracting  officers  might  not  have  a  true  sense  of  buyer 
satisfaction.  Customers  (end-users)  may  not  share  their  satisfaction  or  dissatisfaction  with 
a  Contractor  to  the  contracting  officer  unless  there  are  significant  problems.  If  the 
problems  are  minor  and  can  be  worked  out  with  the  contracting  officer  representatives 
(COR)  or  customer,  the  contracting  officer  might  not  have  knowledge  of  the  situation. 
Secondly,  this  research  was  exploratory.  The  survey  length  was  a  chief  concern,  which 
we  suspect  resulted  in  a  low  response  rate.  Some  of  the  acquisition  strategy  variables 
relied  on  single  indicators  (e.g.,  dissatisfaction  with  discussions,  appropriateness  of 
source  selection  method  and  etc.)  Finally,  the  responses  from  the  survey  were  subjected 
to  the  problems  associated  with  self-reports  in  organizational  research  such  as  common 
method  bias  (Podsakoff  &  Organ,  1986). 

F.  AREAS  FOR  FURTHER  RESEARCH 

This  research  only  focused  on  a  few  factors  associated  with  fear  of  protest. 
According  to  the  models’  low  adjusted  R-square  values;  there  are  other,  unexplored 
factors  that  contribute  to  the  fear  of  protest.  Because  of  time  and  survey  constraints,  this 
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research  was  not  able  to  examine  all  possible  factors.  Therefore,  it  is  recommended  that 
future  research  in  the  below  areas  be  done  to  enhance  our  understanding  of  the  causes 
and  effects  of  a  fear  of  protest. 

1.  Risk  Tolerance 

There  could  be  a  relationship  between  a  person’s  level  of  risk  tolerance  and  the 
level  of  fear.  If  a  person  is  risk-averse,  that  person  will  do  certain  actions  to  reduce  the 
risk.  In  terms  of  a  protest,  a  risk-averse  person  may  have  more  concern  of  a  protest.  It  is 
possible  that  the  risk  averse  person  would  change  the  uncertainty  of  having  a  protest  to 
the  certainty  of  not  having  a  protest.  If  a  person  has  a  high  risk  tolerance,  then  that  person 
is  willing  to  accept  risks.  That  person  might  not  be  as  concerned  about  a  protest.  A 
protest  to  that  person  “is  the  cost  of  doing  business.”  Agencies  should  balance  a  person’s 
level  of  risk  tolerance  against  the  procurement.  Agencies  can  lower  transaction  costs  by 
accepting  more  risk  and  by  assessing  the  risk  tolerance  of  the  team  members  to  ensure 
appropriate  risk  comfort  levels  are  appropriate  for  the  requirement.  If  the  procurement 
has  a  low  risk  of  protest,  a  high-risk  tolerance  person  should  handle  the  procurement.  The 
same  is  true  for  procurement  with  a  high  risk  of  protest;  a  risk-averse  person  should 
handle  the  procurement.  The  risk-averse  person  could  take  the  extra  time  and  resources  to 
ensure  the  source  selection  is  sound. 

2.  Previous  Experience  with  a  Protest 

Fear  is  a  natural  reaction  to  an  unknown  threat.  From  the  data,  contracting  officers 
have  experienced  only  an  average  of  1.96  protests  throughout  their  career.  Many 
contracting  officers  never  experience  a  protest.  Contracting  officers  understand  the 
general  repercussions  of  a  protest  through  press  accounts,  GAO  reports,  and  interaction 
with  those  who  have  experienced  one.  When  contracting  officers  experience  a  protest, 
their  view  of  the  protest  process  can  change.  It  is  a  logical  conclusion  that  if  a  contracting 
officer  has  experience  with  protest,  they  may  be  less  fearful  of  a  protest.  However,  it  is 
possible  that  having  experienced  a  protest,  the  contracting  officer  might  be  more  inclined 
to  alter  acquisition  strategies  to  avoid  the  painful  experience  of  additional  work,  shame, 
and  career  repercussions.  The  implication  is  for  agencies  to  ensure  their  contracting 
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officers  have  personal  experience  in  dealing  with  a  protest.  Requiring  contracting  officers 
to  have  personal  experience  could  be  a  difficult  task  because  the  rational  thought  is  to 
prevent  a  protest  from  happening.  If  an  agency  has  a  protest,  the  contracting  officers  with 
more  experience  of  a  bid  protest  should  actively  mentor  and  engage  less  experience 
contracting  officers  in  the  protest  adjudication  process.  Instead  of  having  solely  the 
experienced  contracting  officer,  or  the  assigned  procuring  contracting  officer,  other 
contracting  personnel  could  be  engaged  in  the  protest  as  well.  This  mentorship  could  be 
structured  as  an  apprenticeship.  Further,  given  the  way  the  internet  can  close  the 
geographic  divide,  apprentice  contracting  personnel  could  be  involved  regardless  of 
physical  separation  from  the  protested  contracting  activity.  The  additional  contracting 
personnel  on  the  protest  adds  additional  cost,  but  the  benefit  of  gaining  firsthand 
knowledge  and  experience  could  outweigh  the  costs  for  the  agency.  Currently, 
government  agencies — more  specifically,  contracting  activities  within  them — generate  a 
plethora  of  source  selection  experience.  However,  that  experience  is  not  systematically 
leveraged  to  more  quickly  build  more  vast  competence  among  the  entire  contracting 
workforce.  Is  there  a  way  to  learn  from  others’  mistakes,  thereby  averting  repeats? 

3.  Inappropriate  Source  Selection  Method 

There  could  be  numerous  reasons  why  source  selection  teams  used  source 
selection  methods  that  were  perceived  to  be  inappropriate.  The  data  shows  that  13  out  of 
133  respondents  reported  that  their  use  of  LPTA  was  to  some  degree  inappropriate. 
Additionally,  10  respondents  out  of  174  reported  that  their  use  of  a  full  trade-off  was  to 
some  degree  inappropriate.  Additional  research  is  needed  to  narrow  down  the  specific 
causes  of  the  inappropriateness  of  the  source  selection  method.  It  could  have  been  solely 
because  of  fear  of  protest,  time  constraints,  guidance  from  senior  management  on  the  best 
source  selection  to  use  for  that  particular  procurement,  or  a  combination  thereof.  There  is 
the  possibility  that  the  contracting  officer’s  personal  perspective  could  have  played  a  role 
in  determining  whether  the  source  selection  used  was  appropriate  or  not.  If  there  are 
systematic  circumstances  on  why  contracting  officers  are  using  LPTA  or  trade-off 


104 


inappropriately,  agencies  should  explore  and  implement  measures  to  reduce  the 
circumstances  that  contribute  to  inappropriate  use.  Possible  reasons  for  inappropriate  use 
are: 

•  To  avoid  a  protest 

•  To  avoid  high  transaction  costs  associated  with  a  full  trade-off 

•  Lack  of  experience  with  full  trade-offs 

•  A  lack  of  understanding  of  the  true  complexity  of  the  requirement 

If  some  of  the  aforementioned  reasons  are  why  inappropriate  source  selections  are 
used,  this  could  be  fuel  for  further  reform  FAR  Part  15.  Perhaps  strict  rules  are 
dissuading  source  selection  teams  from  using  the  most  valuable  sourcing  tool  available  to 
them  (i.e.,  full  trade-off).  Perhaps  the  complexity  of  discussions  causes  source  selection 
teams  to  forego  gaining  further  insight  into  offerors’  proposals. 

4.  Awarding  on  Initial  Proposals 

According  to  Mr.  Gordon,  “Contracting  officers  prefer  to  make  award  based  on 
initial  proposals,  rather  than  to  conduct  discussions,  because  they  fear  that  discussions 
with  offerors  are  a  legal  minefield,  such  that  conducting  discussions  will  increase  the 
likelihood  of  a  bid  protest  and  improve  the  protester’s  chances  of  prevailing  if  a  protest  is 
filed.”  (Gordon,  2013,  p.  37)  This  research  did  not  examine  the  likelihood,  significance, 
or  magnitude  of  the  relationship  between  fear  of  protest  and  award  of  contracts  based  on 
the  initial  proposal.  Awarding  contracts  based  on  initial  proposals  could  have  secondary 
and  tertiary  effects.  There  could  be  a  significant  link  between  fear  of  protest  to  awarding 
contracts  from  initial  proposals.  That  link  could  extend  to  the  source  selection  method  fit, 
and  then  to  the  Contractor  Performance,  which  could  ultimately  have  an  effect  on  buyer’s 
satisfaction.  If  there  is  a  significant  relationship  of  fear  of  protest  and  the  award  of  initial 
proposals,  then  contracting  agencies  should  reduce  the  practice  because  discussions  are 
beneficial  to  the  contractor  and  the  agency,  which  would  be  beneficial  to  the  buyer. 

G.  SUMMARY 

In  conclusion,  fear  of  the  real  or  perceived  consequences  of  receiving  a  bid  protest 

exists.  U.S.  Navy  contracting  officers  have  some  concern  of  protests  due  and  this  can  be 
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linked  to  certain  consequences  on  acquisition  strategies.  However,  the  fear  of  protest 
appears  not  to  be  a  substantial  issue  in  the  Navy.  Across  five  survey  questions,  the 
average  score  was  a  4.08  (just  beyond  the  middle)  on  a  one  to  seven  point  scale.  The 
score  would  indicate  that,  on  average,  the  respondents  fell  between  neither  “disagree  nor 
agree”  and  “somewhat  agree.”  There  are,  however,  instances  where  aspects  of  acquisition 
strategies  are  altered  and  this  behavior  is  statistically  linked  to  the  fear  of  protest.  Given 
the  low  r-squared  values,  however,  factors  other  than  a  fear  of  protest  are  also  culprits. 
Further  research  is  needed  to  ascertain  these  other  culprits,  then  compare  the  relative 
effects  of  fear  of  protest  among  other  factors.  Nonetheless,  there  is  enough  qualitative 
and  quantitative  empirical  evidence  to  suggest  that  fear  of  protest  can  impact  what  would 
otherwise  be  prudent  business  decisions.  The  greatest  concerns  are  a  few  instances  of 
inappropriate  uses  of  LPTA  and  the  reduced  technical  evaluation  effectiveness  attributed 
to  fear  of  protests.  If  fear  waters  down  the  source  selection,  hindering  its  ability  to 
distinguish  between  the  true  value  of  offers,  then  contracting  officers  must  ask 
themselves  why  go  through  the  trouble  of  a  best-value  source  selection?  Could 
contracting  officers  simply  award  to  the  low  bidder?  To  what  extent  is  the  set  of  stringent 
source  selection  rules  driving  the  acquisition  team  to  this  result  by  default  (i.e.,  regardless 
of  source  selection  method  actually  employed)?  Thus,  for  the  sake  of  stringent,  fairness- 
based  rules,  contracted  outcomes  can  be  compromised.  Whether  the  tradeoff  is  prudent 
remains  to  be  determined.  Further  research  is  needed  to  ascertain  these  other  culprits, 
then  compare  the  relative  effects  of  fear  of  protest  among  other  factors. 
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APPENDIX  A 


SURVEY  INSTRUMENT 


NPS  Cmerpriie  Survey  •  fid  Protest 


5/12/14. 10:34  PM 


Bid  Protest 


Ttiere  are  115  questions  in  this  survey 


Consent 


[] 

By  clicking  on  the  "Yes"  button,  I  am  acknowledging  that  I  have  read  and 
understand  this  information,  that  I  understand  the  nature  and  purpose  of  this  study 
-  including  its  risks  and  benefits,  and  that  I  agree  to  voiuntariiy  participate  in  this 
oniine  survey.  I  aiso  understand  that  I  may  discontinue  at  any  time  simpiy  by 
exiting  this  website. 

This  survey  meets  the  requirements  of  OPNAV  5300. 8C  and  does  not  require  a 
formai  RCS  number  since  it  meets  the  criterion  of  a  task  anaiysis.  It  has  been 
reviewed  and  approved  by  the  Navy  Survey  Review  Program  Manager.  * 

Please  choose  only  one  of  the  folowing; 

O  Yes 
O  No 


hnps://wrwev.nps.frdu/<dfnin/prinuMesuivey/M/intfex/iuiveyid/714497 
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NPS  Enterprtsc  Survey  •  SM  Protest 


S/12/14. 10:34  PM 


Group  1 

On  a  scale  of  1  to  7,  where  1  represents  "Strongly  agree"  and  7  represents  "Strongly  disagree",  rate  the  folowtog  statements. 


[] 

Please  refer  back  to  the  most-recently  completed  FAR  Part  15  source  selection 
above  $150K  in  which  you  participated.  Answer  the  questions  in  this  survey 
pertaining  to  this  selected  source  selection. 


On  a  scale  of  1  to  7,  where  1  represents  "Strongly  disagree"  and  7  represents 
"Strongly  agree",  rate  the  following  statements. 

At  some  point  during  the  development  of  the  acquisition  strategy  or  the  source 
selection  process,  I  worried  about  recieving  a  bid  protest.  * 

Please  choose  only  one  of  the  folowing: 

O  1;  Strongly  disagree 
O  2:  Disagree 
O  3:  Somewhat  disagree 
O  4:  Neither  diagree  or  agree 
O  5;  Somewhat  agree 
C)  6:  Agree 
C?  7:  Strongly  agree 


httpi://ui«vev.npi.edu/admif)/priiKable$ufV«y/u/lnitex/Mjrveyid/714497 
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NPS  Enterprise  Survey  >  BM  Protest 


5/12/14, 10:34  PM 


[]I  was  concerned  that  the  contract  award  would  be  protested.  * 
Please  choose  only  one  of  the  folowing; 

Q  1;  Strongly  disagree 
Q  2:  Disagree 
O  3:  Somewhat  disagree 
O  4;  Neither  diagree  or  agree 
O  5;  Somewhat  agree 
O  6;  Agree 
O  7;  Strongly  agree 


[]I  was  anxious  to  get  beyoiid  the  10-day  point  after  contract  award  (or 
debriefings)  to  determine  whether  or  not  the  contract  would  be  protested.  * 

Please  choose  only  one  of  the  folowing: 

C)  1;  Strongly  disagree 
O  2:  Disagree 
O  3:  Somewhat  disagree 
O  4;  Neither  diagree  or  agree 
O  5;  Somewhat  agree 
O  6:  Agree 
O  7:  Strongly  agree 


httpi://siMvey.npS.Mlu/admin  /  printjMe^urvry  .'index  r'vurvryid  .'714497 
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NPS  Enterprise  Survey  •  BM  Protest 


S/12/14. 10:34  PM 


[]Receiving  a  bid  protest  would  have  been  among  the  worst  things  that  could 
happen.  * 

Please  choose  only  one  of  the  folowing: 

O  1;  Strongly  disagree 
O  2;  Disagree 
O  3;  Somewhat  disagree 
O  4:  Neither  diagree  or  agree 
C)  5;  Somewhat  agree 
C)  6;  Agree 
C)  7:  Strongly  agree 


hetpi://iuivey.npi.e<tu/admin/printaMcsurvev/sa/irMlcx/surveyid/714497 
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NPS  EnierpriM  Survey  •  Bid  Protest 


S/12/14, 10:34  PM 


Group  2 

On  a  scate  of  1  to  7,  where  1  represents  "Not  at  all  concemerT  and  7  represents  ‘Extremety  concerned*,  rate  the  followir>g. 


[] 

Please  refer  back  to  the  most-recently  completed  FAR  Part  15  source  selection 
above  $150K  in  which  you  participated.  Answer  the  questions  in  this  survey 
pertaining  to  this  selected  source  selection. 


On  a  scale  of  1  to  7,  where  1  represents  "Not  at  all  concerned"  and  7  represents 
"Extremely  concerned",  rate  the  following. 


During  the  development  of  the  acquisition  strategy  and  throughout  proposal 
evaluation,  to  what  extent  were  you  concerned  that  an  offeror  might  protest  the 
contract  award?  * 

Please  choose  only  one  of  the  folowing: 

C)  1  Not  at  all  concerned 

O  2 

O  3 

C)  4  Concerned 
O  5 

O  6 

O  7  Extremely  concerned 


hitp&://ttM>itv.np».«lu/adfnin/primablesurvey/ia/1n(lex/uirvryid/ 714497 
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<IPS  EnterpriM  Survey  -  Bkf  Proteit 


5/12/14, 10:34  PM 


[] 

During  the  development  of  the  acquisition  strategy  and  throughout  proposal 
evaluation,  to  what  extent  was  at  least  one  other  member  of  the  source  selection 
team  concerned  that  an  offeror  might  protest  the  contract  award?  * 

Please  choose  only  one  of  the  folowing: 

O  1  Not  at  all  concerned 

O  2 

O  3 

O  4  Concerned 
O  5 

O  6 

O  7  Extremely  concerned 


)nps://uHvev.np».edu/admin/prfntablesurvey/u/index/Mirv»vld/714497 
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NPS  Enterprise  Survey  >  BM  Protest 


5/12/14. 10:34  PM 


Group  3 

On  a  scale  of  1  to  7,  where  1  represents  “Strongly  Disagree’  and  7  represents  “Strongly  Agree’,  rate  the  following  statements. 


[] 

Please  refer  back  to  the  most-recently  completed  FAR  Part  15  source  selection 
above  $150K  in  which  you  participated.  Answer  the  questions  in  this  survey 
pertaining  to  this  selected  source  selection. 


On  a  scale  of  1  to  7,  where  1  represents  "Strongly  Disagree"  and  7  represents 
"Strongly  Agree",  rate  the  following  statements. 

I  was  empowered  to  make  required  decisions  throughout  the  source  selection. 
* 

Please  choose  only  one  of  the  folowing: 

O  1;  Strongly  disagree 

O  2:  Disagree 

O  3;  Somewhat  disagree 

O  4:  Neither  dlagree  or  agree 

O  5;  Somewhat  agree 

O  6:  Agree 

C?  7:  Strongly  agree 


hltpi://ui«vev.Api.edu/ddminipfintdMcsurvey/&a/initex/Mifveyid/ 714497 
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NFS  Enterprise  Survey  '  BM  Protest 


5/12/U.  10:34  PM 


[]I  was  trusted  that  the  decisions  I  made  throughout  the  source  selection  would  be 
appropriate.  * 

Please  choose  only  one  of  the  folowing; 

O  1;  Strongly  disagree 
O  2;  Disagree 
O  3:  Somewhat  disagree 
O  4:  Neither  diagree  or  agree 
C?  5;  Somewhat  agree 
O  6:  Agree 
C)  7:  Strongly  agree 


[] 

My  management  supported  me  on  the  decisions  I  made  during  the  source  selection. 
« 

Please  choose  only  one  of  the  folowing; 

C>  1;  Strongly  disagree 
O  2;  Disagree 
O  3:  Somewhat  disagree 
O  4:  Neither  diagree  or  agree 
O  5:  Somewhat  agree 
O  6:  Agree 
O  7;  Strongly  agree 


hnpi://Miryev.npi.edu/adfnin/pr{maMe4uryey/u/ii>(lej(yMirveyid/714497 
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NPS  EntcrpriM  Survey  •  BM  Protest 


S/12/14. 10:34  PM 


[] 

If  I  disagreed  with  an  aspect  of  a  legal  opinion/review,  I  had  the  latitude  to  deviate 
from  it.  ♦ 

Please  choose  only  one  of  the  folowing; 

C)  1:  Strongly  disagree 
O  2;  Disagree 
O  3;  Somewhat  disagree 
O  4;  Neither  diagree  or  agree 
O  5;  Somewhat  agree 
O  6:  Agree 
O  7:  Strongly  agree 


[] 

I  had  to  change  documents  generated  during  the  source  selection  to  correspond 
with  reviewers.  * 

Please  choose  only  one  of  the  folowing; 

O  1;  Strongly  disagree 
O  2;  Disagree 
C)  3;  Somewhat  disagree 
C)  4:  Neither  diagree  or  agree 
O  5:  Somewhat  agree 
O  6;  Agree 
O  7;  Strongly  agree 


http«://Mjrvev.nps.edu/«dinin/pdnt^)ie»uryey/u/intfex/suryeyid/714497 


P«9e  9  54 


115 


NFS  enterprise  Survey  •  Sd  Protest 


S/12/14.  10:34  PM 


[] 

I  might  as  well  not  have  a  warrant  since  my  decisions  were  overridden  by 
reviewers.  ♦ 

Please  choose  only  one  of  the  folowing; 

C?  1:  Strongly  disagree 
O  2:  Disagree 
O  3:  Somewhat  disagree 
O  4;  Neither  diagree  or  agree 
O  5;  Somewhat  agree 
O  6;  Agree 
O  7;  Strongly  agree 


https://stMV«v.nps.odu/admin/pr1ntat)lesurvey/s«/irMlex/surMyid/714497 
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NPS  Enterprise  Survey  •  Sd  Protest 


S/12/14, 10:34  PM 


Group  5 

On  a  scale  of  1  to  7,  where  1  represents  "Slrongty  Disagree'  and  7  represents  "Strongly  Agree",  rate  the  following  statements. 


[] 

Please  refer  back  to  the  most-recently  completed  FAR  Part  15  source  selection 
above  $150K  in  which  you  participated.  Answer  the  questions  in  this  survey 
pertaining  to  this  selected  source  selection. 


On  a  scaie  of  1  to  7,  where  1  represents  "Strongiy  Disagree"  and  7  represents 
"Strongly  Agree",  rate  the  following  statements. 

The  milestones  for  awarding  this  contract  were  too  aggressive.  * 

Please  choose  only  one  of  the  folowing; 

C)  1 ;  Strongly  disagree 
O  2:  Disagree 
C)  3:  Somewhat  disagree 
O  4:  Neither  disagree  or  agree 
O  5:  Somewhat  agree 
O  6:  Agree 
O  7;  Strongly  agree 


hnp»://»uivev.nps.edu/«lmin/prinuMesurv«y/u/ltM)ex/$urveyid/714497 
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NPS  Enterprise  Survey  •  M  Protest 


5/12/14, 10:34  PM 


[]I  was  not  rushed  to  award  this  contract.  * 
Please  choose  only  one  of  the  folowing; 

O  1 :  Strongly  disagree 
C)  2:  Disagree 
O  3:  Somewhat  disagree 
O  4:  Neither  disagree  or  agree 
O  5:  Somewhat  agree 
O  6:  Agree 
O  7:  Strongly  agree 


[] 

I  had  sufficient  time  to  get  this  contract  awarded.  * 
Please  choose  only  one  of  the  folowing; 

C)  1:  Strongly  disagree 
O  2;  Disagree 
O  3:  Somewhat  disagree 
O  4:  Neither  disagree  or  agree 
O  5;  Somewhat  agree 
O  6;  Agree 
C)  7:  Strongly  agree 


hnps://$u4vcy.nps.edu/Mknin/printaMesurvev/ia/in(lex/surv«vid/714497 
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NPS  CmerprtM  Survey  >  BM  Protest 


5/12/14. 10:34  PM 


[]At  least  once,  a  technical  evaluator  was  required  to  change  the  wording  of  their 
technical  evaluations.  * 

Please  choose  only  one  of  the  folowing; 

O  1:  Strongly  disagree 

O  2;  Disagree 

O  3:  Somewhat  disagree 

O  4;  Neither  disagree  or  agree 

C?  5:  Somewhat  agree 

C)  6:  Agree 

C)  7;  Strongly  agree 


[] 

At  least  one  technical  evaluator  expressed  concern  about  not  being  able  to  say  what 
needs  to  be  said  in  the  technical  evaluation.  * 

Please  choose  only  one  of  the  folowng: 

C)  1:  Strongly  disagree 

O  2;  Disagree 

C)  3:  Somewhat  disagree 

O  4:  Neither  disagree  or  agree 

O  5:  Somewhat  agree 

O  6:  Agree 

O  7;  Strongly  agree 


hctp&://uMvev.nps.edu/admjn/p«1ncaMeswfvey/S4/ifMlex/Mirvev<d/714497 
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NPS  Enterprise  Survey  -  BM  Protest 


S/12/14, 10:34  PM 


[] 

At  least  one  technical  evaluator  was  concerned  that  the  constraints  imposed  on 
their  evaluations  impeded  his/her  ability  to  write  a  meaningful  evaluation.  * 

Please  choose  only  one  of  the  folowing: 

O  1;  Strongly  disagree 
O  2:  Disagree 
CD  3:  Somewhat  disagree 
CD  4;  Neither  disagree  or  agree 
O  5;  Somewhat  agree 
O  6;  Agree 
O  7;  Strongly  agree 


[] 

The  technical  evaluators  believed  that  the  quality  of  their  evaluations  could  not 
have  been  better.  * 

Please  choose  only  one  of  the  folowing; 

O  1:  Strongly  disagree 
CD  2:  Disagree 
CD  3:  Somewhat  disagree 
CD  4;  Neither  disagree  or  agree 
CD  5;  Somewhat  agree 
CD  6;  Agree 
O  7;  Strongly  agree 


hitps://Mirvev.nps.edu/admin/pr1nubleMfvev/M/in(lex/$uiwyid/ 714497 
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NPS  Enicrpri$e  Survey  •  Bkl  Protest 


S/12/14. 10:34  PM 


[] 

If  there  were  no  Federal  Acquisition  Regulations,  no  source  selection  policy,  and  no 
threat  of  a  bid  protest,  the  quality  of  the  technical  evaluations  would  have  been  the 
same.  * 

Please  choose  only  one  of  the  folowing; 

O  1;  Strongly  disagree 
O  2;  Disagree 
O  3:  Somewhat  disagree 
O  4:  Neither  disagree  or  agree 
C)  5;  Somewhat  agree 
C)  6;  Agree 
C)  7:  Strongly  agree 


[]Upon  evaluation  of  proposals,  at  least  one  technical  evaluator  expressed  a  need  to 
change  at  least  one  evaluation  criterion  or  its  definition.  * 

Please  choose  only  one  of  the  folowing; 

O  1 :  Strongly  disagree 
O  2;  Disagree 
O  3:  Somewhat  disagree 
C)  4;  Neither  disagree  or  agree 
C)  5;  Somewhat  agree 
O  6:  Agree 
O  7:  Strongly  agree 


hnpi://M»fvey.np».edu/Mtfnjn/pcimaMesurv«y/u/fn(lex/$urveyid/714497 
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NPS  CnicrpriM  Survey  -  Sd  Protest 


S/12/14.  10:34  PM 


[]As  a  portion  of  the  customer's  total  annual  spending  amount,  the  dollar  value  of 
this  requirement  was  high.  * 

Please  choose  only  one  of  the  folowing; 

O  1;  Strongly  disagree 
O  2:  Disagree 
O  3;  Somewhat  disagree 
O  4;  Neither  diagree  or  agree 
O  5;  Somewhat  agree 
O  6:  Agree 
C)  7;  Strongly  agree 


[] 

This  requirement  was  important  for  the  good  operation  of  our  customer's 
organization. 

Please  choose  only  one  of  the  folowing; 

C)  1:  Strongly  disagree 
O  2:  Disagree 
O  3:  Somewhat  disagree 
O  4;  Neither  disagree  or  agree 
O  5;  Somewhat  sigree 
O  6;  Agree 
O  7:  Strongly  agree 


httpi://stMvev.npi.edu/Mtfnin/prim«MeMrvry/u/iAdex/iurvryid/714497 
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NPS  EmerpriM  Survey  •  Bkl  Protest 


5/12/14,  10:34  PM 


[] 

This  requirement  supported  a  core  competency  of  our  customer's  organization.  * 
Please  choose  only  one  of  the  folowing: 

O  1:  Strongly  disagree 

O  2;  Disagree 

C)  3:  Somewhat  disagree 

C)  4:  Neither  disagree  or  agree 

C?  5;  Somewhat  agree 

C)  6:  Agree 

C?  7:  Strongly  agree 


[] 

Compared  to  other  purchases  for  this  customer,  this  requirement  was  important.  * 
Please  choose  only  one  of  the  folowing; 

C?  1;  Strongly  disagree 
C)  2;  Disagree 
C)  3:  Somewhat  disagree 
C)  4:  Neither  disagree  or  agree 
O  5:  Somewhat  agree 
O  6;  Agree 
O  7:  Strongly  agree 


http&://uirvey.npi.e<Su/adfnin/pflmaMesuryey/u/intfex/$urvevid/714497 
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NPS  CmcrpriM  Survey  •  BM  Protest 


S/12/14. 10:34  PM 


[] 

An  unsuccessful  outcome  of  the  RFP  would  have  had  only  minor  consequences  to 
our  customer.  * 

Please  choose  only  one  oT  the  folowing: 

O  1 :  Strongly  disagree 
C)  2:  Disagree 
O  3:  Somewhat  disagree 
O  4:  Neither  disagree  or  agree 
O  5:  Somewhat  agree 
O  6;  Agree 
O  7:  Strongly  agree 


https://survev.nps.edu/Mlmin/pfinubiesurvey/uAirHlex/survevid/714497 
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NPS  Enierprise  Survey  •  M  Protest 


5/12/14, 10:34  PM 


Group  6a 

The  next  five  questions  refer  to  the  contractor  that  was  awarded  a  contract  resulting  from  the  selected  source  selection.  For  a 
multiple-award  IDIQ,  select  the  corrtractor  that  received  the  first  deliveryAask  order  award  arxl  answer  with  respect  to  that 
deliveryrtask  order. 

On  a  scale  of  1  to  7.  where  1  represents  "Strongly  Disagiee'  and  7  represents  "Strongly  Agree",  rate  the  following  statements. 


[] 

Please  refer  back  to  the  most-recently  completed  FAR  Part  15  source  selection 
above  $150K  in  which  you  participated.  Answer  the  questions  in  this  survey 
pertaining  to  this  selected  source  selection. 


The  next  five  questions  refer  to  the  contractor  that  was  awarded  a  contract 
resulting  from  the  selected  source  selection.  For  a  multiple-award  IDIQ,  select  the 
contractor  that  received  the  first  delivery/task  order  award  and  answer  with 
respect  to  that  delivery/task  order.  On  a  scale  of  1  to  7,  where  1  represents 
"Strongly  Disagree"  and  7  represents  "Strongly  Agree",  rate  the  following 
statements. 


Our  customer  regrets  the  decision  to  do  business  with  this  contractor.  * 
Please  choose  only  one  of  the  folowing: 

C)  1:  Strongly  disagree 
O  2:  Disagree 
O  3:  Somewhat  disagree 
O  4:  Neither  diagree  or  agree 
O  5;  Somewhat  agree 
O  6:  Agree 
O  7:  Strongly  agree 


hctps://U)fvey.np&.edu/a<knin/print&Mc»urvey/u/tmlex/&urvey4d/714497 
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NFS  EnicrpriM  Survey  -  Bd  Protect 


5/12/14. 10:34  PM 


[]Overall,  our  custonter  is  very  satisfied  with  this  contractor. 
Please  choose  only  one  of  the  folowing: 

C)  1 :  Strongly  disagree 
O  2:  Disagree 
O  3:  Somewhat  disagree 
O  4:  Neither  diagree  or  agree 
O  5:  Somewhat  agree 
O  6;  Agree 
O  7;  Strongly  agree 


[] 

Our  customer  is  very  pleased  with  what  the  contractor  does  for  them.  * 
Please  choose  only  one  of  the  folowing; 

O  1 :  Strongly  disagree 
O  2;  Disagree 
O  3:  Somewhat  disagree 
O  4:  Neither  diagree  or  agree 
O  5;  Somewhat  agree 
O  6:  Agree 
O  7;  Strongly  agree 


hnpi://uirvey.nps.edu/adfnin/prinuMeMifvry/u/intfex/ujiveyid/714497 
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NPS  EfilcrpriM  Survey  •  BM  Protest 


S/12/14. 10:34  PM 


[] 

Our  customer  is  not  completely  happy  with  this  contractor.  * 
Please  choose  only  one  of  the  folowing; 

O  1;  Strongly  disagree 
O  2:  Disagree 
O  3:  Somewhat  disagree 
O  4:  Neither  diagree  or  agree 
O  5;  Somewhat  agree 
O  6;  Agree 
O  7;  Strongly  agree 


[] 

If  we  had  to  do  it  all  over  again,  we  would  still  choose  to  use  this  contractor.  * 
Please  choose  only  one  of  the  folowing; 

C>  1:  Strongly  disagree 
C)  2:  Disagree 
O  3;  Somewhat  disagree 
C)  4:  Neither  diagree  or  agree 
CD  5:  Somewhat  agree 
CD  6:  Agree 
O  7;  Strongly  agree 
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Group  6b 


The  next  seven  questions  refer  to  the  contractor  and  contract  or  delivery/task  order  used  to  answer  the  previous  five  questions.  On 
a  scale  of  1  to  7.  where  1  represents  'needs  improvemefTt'  and  7  represents  ‘superior  performance',  rate  the  following. 


[] 

The  next  seven  questions  refer  to  the  contractor  and  contract  or  delivery/task  order 
used  to  answer  the  previous  five  questions.  On  a  scale  of  1  to  7,  where  1 
represents  "needs  improvement"  and  7  represents  "superior  performance",  rate  the 
following. 


Product/service  quality  per  specifications  * 
Please  choose  only  one  of  the  folowing: 

O  1  Needs  improvement 

O  2 

O  3 

O  4  Adequate  performance 
O  5 

O  6 

C)  7  Superior  Performance 


[]Delivery  performance  per  specifications  * 
Please  choose  only  one  of  the  folowing: 

O  1  Needs  improvement 

O  2 

O  3 

O  4  Adequate  performance 
O  5 

O  6 

O  7  Superior  Performance 
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[]Required  service  and/or  technical  support  * 
Please  choose  only  one  of  the  folowing; 

C)  1  Needs  improvefnent 

O  2 

O  3 

O  4  Adequate  performance 
O  5 

O  6 

O  7  Superior  Performance 


[] Non-conformance  rate  * 
Please  choose  only  one  of  the  folowing; 

O  1  Needs  improvement 

O  2 

O  3 

O  4  Adequate  performance 
O  5 

O  6 

O  7  Superior  Performance 


[jOverall  performance  * 
Please  choose  only  one  of  the  folowing; 
O  f  Needs  improvement 

O  2 

O  3 

O  4  Adequate  perfomnance 
O  5 

O  6 

O  7  Superior  Performance 
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Group  7 

On  a  scale  of  -5  to  5,  where  -5  represents  ‘completely  undesirable'  and  5  represents  ‘completely  desirable,*  please  rate  the  extent 
of  desirabitty  associated  with  each  possfcle  consequence  of  a  bid  protest. 


[] 

On  a  scale  of  -5  to  5,  where  -5  represents  "completely  undesirable"  and  S 
represents  "completely  desirable,"  please  rate  the  extent  of  desirability  associated 
with  each  possible  consequence  of  a  bid  protest. 


Increased  costs  to  settle  a  terminated  contract(s).  * 

Please  choose  only  one  of  the  folowing; 

C)  -5  Completely  undesirable 

o 

O  -3 

O  -2 
O  -1 

O  0  Does  not  matter 

O  1 
O  2 

O  3 
O  4 

O  5  Completely  desirable 
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[]Time  delay  to  the  mission.  * 
Please  choose  only  one  of  the  folowing; 

C)  -5  Completely  undesirable 
O  -4 
D  -3 
O  -2 
O  -1 

O  0  Does  not  matter 

O  1 
O  2 

O  3 
O  4 

O  5  Completely  desirable 


[] 

Enibarrassment/shame.  * 
Please  choose  only  one  of  the  folowing; 

O  -5  Completely  undesirable 

O  -I 
O  -3 
-2 
o  -1 

C)  0  Does  not  matter 

O  1 
O  2 

O  3 
O  4 

O  5  Completely  desirable 
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[]Increase  in  workload  to  resolve  the  protest.  * 
Please  choose  only  one  of  the  folowing: 

O  -5  Completely  undesirable 

O  -I 

O  -3 

O  -2 
O  -1 

O  0  Does  not  matter 

O  1 
O  2 

O  3 
O  4 

C)  5  Completely  desirable 


[]Career  repercussions  for  making  a  mistake  or  omission  that  caused  a  bid  protest. 
* 

Please  choose  only  one  of  Ihe  folowing: 

O  -5  Completely  undesirable 

O  -1 

O  -3 

O  -2 
O  -1 

C)  0  Does  not  matter 

O  1 

O  2 

O  3 
O  4 

O  5  Completely  desirable 
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Group  8 

In  the  event  of  a  bid  protest  please  rate  the  probability  that  each  consequence  might  occur. 


[] 

In  the  event  of  a  bid  protest  please  rate  the  probability  that  each  consequence 
might  occur. 

Increased  costs  to  settle  a  terminated  contract(s).  * 

Ptease  choose  only  one  of  the  folowing; 


o 

0% 

o 

10% 

o 

20% 

o 

30% 

o 

40% 

o 

50% 

o 

60% 

o 

70% 

o 

80% 

o 

90% 

o 

100% 
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[]Time  delay  to  the  mission.  * 

Please  choose  only  one  of  the  folowing: 

o 

0% 

o 

10% 

o 

20% 

o 

30% 

o 

40% 

o 

50% 

o 

60% 

o 

70% 

o 

80% 

o 

90% 

o 

100% 

[]Embarrassment/shame.  * 
Please  choose  only  one  of  the  folowing: 

C>  0% 

O  10% 

O  20% 

O  30% 

O  40% 

O  50% 

O  60% 

O  70% 

O  80% 

□  90% 

O  100% 
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Group  9 

On  a  scale  o(  1  to  7,  where  1  represents  ‘Strongly  Disagree’  and  7  represents  ‘Strongly  Agree’,  rate  the  following  statements. 


[] 

Please  refer  back  to  the  most-recently  completed  FAR  Part  15  source  selection 
above  $1S0K  in  which  you  participated.  Answer  the  questions  in  this  survey 
pertaining  to  this  selected  source  selection. 


On  a  scale  of  1  to  7,  where  1  represents  "Strongly  Disagree"  and  7  represents 
"Strongly  Agree",  rate  the  following  statements. 

Our  acquisition  strategy  was  the  best  means  to  source  our  requirentent.  * 
Please  choose  only  one  of  the  folowing; 

O  1 :  Strongly  disagree 
C)  2:  Disagree 
O  3:  Somewhat  disagree 
O  4:  Neither  diagree  or  agree 
O  5:  Somewhat  agree 
O  6;  Agree 
O  7:  Strongly  agree 


hitp$://uMvev.npi.edu/adniin/princali4esuivey/&A/index/&urveyid/714497 


Pipe  31  of  54 


137 


NPS  Enterprise  Survey  -  BM  Protest 
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[]Our  acquisition  strategy  was  the  best  means  to  achieve  our  acquisition  objectives. 
* 

Please  choose  only  one  of  the  folowing; 

O  1:  Strongly  disagree 
O  2:  Disagree 
O  3;  Somewhat  disagree 
O  4:  Neither  diagree  or  agree 
O  5;  Somewhat  agree 
C)  6:  Agree 
C)  7:  Strongly  agree 


[] 

It  would  have  been  difficult  to  achieve  our  goals  without  the  use  of  our  acquisition 
strategy.  * 

Please  choose  only  one  of  the  folowing; 

C)  1:  Strongly  disagree 
CD  2:  Disagree 
CD  3:  Somewhat  disagree 
O  4:  Neither  diagree  or  agree 
O  5:  Somewhat  agree 
O  6;  Agree 
O  7;  Strongly  agree 
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[] 

The  source  selection  method  we  used  (i.e.,  Low-Price-Technically  Acceptable;  full- 
tradeoff;  or  Price-Past  Performance  Tradeoff)  was  the  most  appropriate  for  this 
requirement.  * 

Please  choose  only  one  of  the  folowing; 

O  1:  Strongly  disagree 
O  2;  Disagree 
O  3:  Somewtiat  disagree 
O  4:  Neither  diagree  or  agree 
C?  5:  Somewhat  agree 
O  6:  Agree 
C)  7:  Strongly  agree 


[] 

Our  acquisition  strategy  ensured  we  selected  the  optimal  offeror.  * 
Please  choose  only  one  of  the  folowing: 

O  1:  Strongly  disagree 
2:  Disagree 

C)  3:  Somewhat  disagree 
C?  4;  Neither  diagree  or  agree 
C)  5;  Somewhat  agree 
O  6:  Agree 
C)  7:  Strongly  agree 
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[] 

Our  acquisition  strategy  provided  the  best  fit  to  the  buying  situation  (e.g., 
complexity,  dollar  value,  acquisition  objectives,  contract  length,  performance  risk, 
criticality  to  the  mission,  availability  of  supply,  time  available  to  award  a  contract, 
etc.).  * 

Ptease  choose  only  one  of  the  folowing; 

C)  1;  Strongly  disagree 
C)  2:  Disagree 
C?  3:  Somewhat  disagree 
C)  4;  Neither  diagree  or  agree 
O  5;  Somewhat  agree 
C)  6;  Agree 
O  7;  Strongly  agree 
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Group  10 


[] 

Please  refer  back  to  the  most-recently  completed  FAR  Part  15  source  selection 
above  $150K  in  which  you  participated.  Answer  the  questions  in  this  survey 
pertaining  to  this  selected  source  selection. 


During  acquisition  planning,  how  many  days  were  planned  in  the  milestones  from 
receipt  of  a  complete  requirements  package  to  award  of  the  contract(s):  * 

Please  write  your  answer  here: 


[]Number  of  actual  days  from  receipt  of  a  complete  requirements  package  to  award 
of  the  contract  * 

Please  write  your  answer  here: 


[]Nuniber  of  people  on  the  source  selection  team  including  all  advisors,  reviewers, 
full-time  participants,  and  part-time  participants:  * 

Please  write  your  answer  here: 


[]Absent  a  risk  of  a  bid  protest,  in  your  opinion,  how  many  people  ideally  should 
have  been  on  the  source  selection  team:  * 

Please  write  your  answer  here; 
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[]What  source  selection  method  was  used?  (Select  one.)  * 
Please  choose  only  one  of  the  folowing: 

O  Low-Price-Technically-Acceptable 
C)  Price-Past  Performance  Tradeoff 
O  Full  tradeoff 


[] 

Rate  the  appropriateness  of  each  source  selection  method  for  the  requirement  on  a 
scale  of  1  to  7,  where  1  represents  "completely  inappropriate"  and  7  represents 
"completely  appropriate."  Appropriate,  in  this  context,  means  the  source  selection 
method  is  the  best  fit  to  the  buying  situation  (e.g.,  complexity,  dollar  value, 
acquisition  objectives,  contract  length,  performance  risk,  criticality  to  the  mission, 
availability  of  supply,  time  available  to  award  a  contract,  etc.) 


LPTA  (Lowest  Price  Technically  Acceptable)  * 
Please  choose  only  one  of  the  fokwing: 

O  1;  Completely  Inappropriate 

O  2; 

O  3: 

O  4: 

O  5: 

O  6; 

O  7:  Completely  appropriate 
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[]PPT  (Price-Past  Performance  Tradeoff  )  * 
Please  choose  only  one  of  the  folowing; 

C)  1:  Completely  inappropriate 

O  2: 

O  3: 

O  4; 

O  5: 

O  6: 

O  7;  Completely  appropriate 


[]Full  Tradeoff  * 

Please  choose  only  one  of  the  folowing: 

O  1:  Completely  inappropriate 

O  2: 

O  3: 

O  4: 

O  5: 

O  6: 

O  7;  Completely  appropriate 
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[] 

On  a  scale  of  1  to  7,  where  1  represents  "completely  satisfied"  and  7  represents 
"completely  dissatisfied,"  rate  your  level  of  satisfaction  with  the  freedom  to  openly 
discuss  those  aspects  of  the  proposals  that  needed  to  be  discussed  with  the  offeror 
in  order  to  fully  understand  the  offer  and  to  properly  evaluate  the  proposals.  * 

Please  choose  only  one  of  the  folowing; 

C)  1;  Completely  satisfied 

O  2: 

O  3: 

O  4; 

O  5: 

O  6: 

O  7;  Completely  dissatisfied 


[]Hypothetically  if  there  was  no  ability  to  protest,  rate  the  extent  that  discussions 
with  offerors  would  have  differed,  on  a  scale  of  1  to  7,  where  1  represents  "no 
difference,"  and  7  represents  "substantial  difference."  * 

Please  choose  only  one  of  the  folowing: 

C)  1  No  difference 

O  2 

O  3 
O  4 
O  5 

O  6 

C)  7  substantial  difference 


[]Did  the  RFP  advise  offerors  that  the  Government  intended  to  award  without 
discussions?  * 

Please  choose  only  one  of  the  folowing: 

O  Yes 
O  No 
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[]Did  you  conduct  discussions?  * 
Please  choose  only  one  of  the  folowing; 

C?  Yes 
C)  No 


[] 

On  a  scale  of  1  to  7,  where  1  represents  "completely  inappropriate"  and  7 
represents  "completely  appropriate,"  considering  the  risk,  criticality,  dollar  value, 
contribution  to  the  mission,  and  complexity,  rate  the  appropriateness  of  awarding  a 
contract  without  conducting  discussions.  [If  you  intended  to  conduct  discussions, 
select  1].  ♦ 

Please  choose  only  one  of  the  folowing; 

O  1:  Completely  Inappropriate 

O  2: 

O  3: 

O  4; 

O  5: 

O  6; 

C?  7:  Completely  appropriate 


[]Did  offerors  make  oral  presentations?  * 
Please  choose  only  one  of  the  folowing; 

C)  Yes 
O  No 
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[] 

On  a  scale  of  1  to  7,  where  1  represents  "completely  inappropriate"  and  7 
represents  "compieteiy  appropriate,"  considering  the  risk,  criticality,  doiiar  vaiue, 
contribution  to  the  mission,  and  compiexity,  rate  the  appropriateness  of  not 
utiiizing  oral  presentations.  [If  offerors  made  orai  presentations,  seiect  1].  * 

Please  choose  only  one  of  the  folowing; 

C)  1:  Completely  Inappropriate 

O  2: 

O  3: 

O  4; 

O  5: 

O  6: 

O  7:  Completely  appropriate 


[]According  to  your  acquisition  pianning,  how  many  contracts  were  originaiiy 
intended  to  be  awarded? 

Please  write  your  answer  here; 


[]How  many  contracts  were  actually  awarded?  * 
Please  write  your  answer  here; 
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Group  10a 

During  the  source  selection,  how  many  iterations  of  each  of  the  following  documents  were  generated? 


[] 

Please  refer  back  to  the  most-recently  completed  FAR  Part  15  source  selection 
above  $150K  in  which  you  participated.  Answer  the  questions  in  this  survey 
pertaining  to  this  selected  source  selection. 


During  the  source  selection,  how  many  iterations  of  each  of  the  following 
documents  were  generated? 

Source  selection  decision  document  * 

Please  write  your  answer  here: 


[]Coniparative  assessment/proposal  analysis  report  * 
Please  write  your  answer  here: 


[]Evaluation  notices  (sum  of  iterations  for  all  ENs  across  all  offerors)  * 
Please  write  your  answer  here: 
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[]Source  selection  plan  * 
Please  write  your  arrswer  here: 


[]Debriefing  scripts  (sum  of  iterations  for  all  scripts  across  all  offerors)  * 
Please  write  your  answer  here; 


[]Technical  evaluations  (sum  of  iterations  for  all  evaluations  across  all  offerors)  * 
Please  write  your  arrswer  here; 


[]Past  performance  evaluations  (sum  of  iterations  for  all  evaluations  across  all 
offerors)  * 

Please  write  your  answer  here; 


[]Cost/Price  analysis  ♦ 
Please  write  your  answer  here; 


[]Color/rating  chart  * 
Please  write  your  answer  here; 


hitpi://utrwev-nps.edu/admin/primaMe4urvev/u/iiKlex/Mii^yid/714497 


P*9e  42  ol  S4 


148 


NPS  EnterprfM  Survey  •  BM  Protest 


S/12/14. 10:34  PM 


[] 

Evaluation  briefing  charts  for  reviewers  and  SSA  * 

Please  write  your  arrswer  here: 
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Group  11 
[] 

Please  refer  back  to  the  most-recently  completed  FAR  Part  15  source  selection 
above  $150K  in  which  you  participated.  Answer  the  questions  in  this  survey 
pertaining  to  this  selected  source  selection. 

Was  the  contract  award  protested?  * 

Please  choose  only  one  of  the  folowing; 

O  Yes 
O  No 


[] 

What  was  the  contract  type?  * 
Please  choose  only  one  of  the  folowing: 

C?  Fixed  Price 
C)  Cost  Reimbursement 
O  Time  and  Materials 
O  Labor-Hour 

O  Hybrid  (multiple  contract  types) 

O  Other 


[]What  was  the  total  dollar  value  of  the  contract/task  order/delivery  order 
(including  all  options)?  * 

Please  write  your  answer  here: 
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[]Was  the  requirement  set  aside  for  Small  Business?  * 
Please  choose  only  one  of  the  folowing; 

C)  Yes 
□  No 


[] 

What  type  of  supply  or  service  is  the  contractor  providing?  * 
Please  choose  only  one  of  the  folowing: 

C)  Service 
C)  Coostnjctkxi 

C)  Supplies/Coftimodities/Spares 
C)  Weapon  System 
O  Other  Capital  Equipment 


[]For  this  contract/task  order/delivery  order,  what  was  the  Product  Service 
Code/Federal  Supply  Class  code?  This  group  code  is  the  first  digit(s)  of  the 
PSC/FSC  code. 

Please  write  your  answer  here: 


https://turvev.np$^u/«daiin/princaMeMrvey/u/index/Mir^yid/714497 
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NPS  Enterprise  Survey  •  Bid  Protest 


S/12/14.  10:34  PM 


Group  12 


[] 

ALMOST  DONE!!! 


Please  refer  back  to  the  most  recently  completed  FAR  part  15  source  selection 
above  150K  in  which  you  participated.  Answer  the  questions  pertaining  to  this 
source  selection. 


For  the  following  roles  please  provide  the  number  of  people  within  that  role,  their 
Grade(s)/Rank(s),  and  the  fraction  of  1  year's  time  and  effort  dedicated  to  the 
Source  Selecrtion.  If  no  involvement,  just  skip  to  the  next  role. 

Format:  (#  of  People;Grade;Fraction  of  one  year's  time) 


Example  #1 

For  example,  if  there  were  3  contracting  officers  involved,  and  a  GS  13  spent  3 
months  of  their  time  to  the  source  selection  and  two  GS  12's  spent  4  months  of 
their  time,  the  appropriate  response  would  be: 

(3;  GS13,GS12,GS12;  25%,33<Vfe,33%) 

Example  #2 

If  only  one  GS  12  was  involved  and  spent  6  months  on  the  source  selection,  the 
appropriate  response  would  be: 

(1;  GS12;  50%) 

Example  #3 

If  one  technical  evaluator  (GS12)  allocated  half  of  his/her  time  to  the  source 
selection  for  6  months,  insert  25%  [=  (.5  year)*(.5  time/ effort)].  The  appropriate 
response  would  be: 


(1;  GS12;  25%) 


hnp«://suivev.nps.edu/admin/printdMesufV«y/u/index/surveyld/714497 
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NPS  Emerpdte  Survey  •  Bkl  Protest 


5/12/14. 10:34  PM 


CONTRACTING  OFFICER 
Ptease  write  your  answer  here; 


[]CONTRACT  SPECIALISTS 
Please  write  your  answer  here: 


[]TECHNICAL  EVALUATORS 
Please  write  your  answer  here; 


[]COST/PRICE  ANALYSTS 
Please  write  your  answer  here: 


[]SUPERVISORS 
Please  write  your  answer  here: 


[]LEGAL 

Please  write  your  answer  here; 


hitpi:  /  ruKvev-npi-edu/Mlmjn/printAlileiurvcy/u  /ifMlexyMirv«yid/  714  497 


P491  47  o(  S4 
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NPS  Cracrpri&e  Survey  •  B&d  Protest 


S/12/14, 10:34  PM 


[]PAST  PERFORMANCE  TEAM 
Please  write  your  answer  here: 


[] 

SOURCE  SELECTION  ADVISORY  COUNCIL/NON-LEGAL  ADVISORS 

Please  write  your  answer  here: 


[] 

SOURCE  SELECTION  AUTHORITY 


Please  write  your  answer  here: 


[] 

FINANCIAL  MANAGEMENT 

Please  write  your  answer  here: 


[] 

PROGRAM  MANAGER/REQUIREMENTS  OFFICE 

Please  write  your  answer  here: 


https://ujrvev.nps.etfu/«d<nin/pdntaMesuryey/u/in(lex/survevid/714497 


P*9e  48  ol  S4 


154 


NPS  EnterpriM  Survey  *  8M  Protect 


5/12/14, 10:34  PM 


[] 

SMALL  BUSINESS  REPRESENTATIVE 

Ptease  write  your  answer  here; 


[]CONTRACTED  CONSULTANTS/LABOR 
Please  write  your  answer  here; 


[] 

Please  write  your  answeits)  here; 

OTHER  (DESCRIPTION) 

(#  of  People;Gracle;Fraction  of  one  year's 
time) 


hnpi:  /  /  Mfv«v.npi.etfu/«dmirt/prln(aMesurycv/u  /Imtex/Mirvcyid/  714497 
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NPS  enterprise  Survey  •  Bkf  Protest 


5/12/14. 10:34  PM 


Group  13 

DEMOGRAPHICS 
LAST  PAGE!!!! 


[] 

LAST  PAGE!! 


General  questions  and  demographics.  Please  answer  the  following  questions 
accordingly. 


Throughout  your  career,  how  many  times  have  you  awarded  a  task  order/delivery 
order  against  an  IDIQ  contract  (or  a  call  against  a  blanket  purchase  agreement)  in 
order  to  avoid  a  protestable  competitive  procurement?  * 

Please  write  your  answer  here; 


[] 

Throughout  your  career,  how  many  times  have  you  sole  sourced  a  requirement  in 
order  to  avoid  a  protestable  competitive  procurement?  * 

Please  write  your  answer  here; 


[]Throughout  your  career,  how  many  times  have  you  found  a  way  to  modify  an 
existing  contract,  order,  call,  or  agreement  in  order  to  avoid  a  protestable 
competitive  procurement?  * 

Please  write  your  answer  here: 


httpi:  // vMVcv.npi.edu/admin/  printaMesurvey/ia  /index/iurveyid/  714497 
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NPS  Enicrprise  Survey  •  BM  Protest 


5/12/14. 10:34  PM 


[]Throiighout  your  career,  how  many  bid  protests  have  you  experienced?  * 
Ptease  write  your  answer  here; 


[]Throughout  your  career,  have  you  ever  experienced  a  sustained  bid  protest?  * 
Please  choose  only  one  of  the  folowing: 

O  Yes 
O  No 


[] 

How  many  totai  years  of  experience  do  you  have  in  contracting?  * 
Please  write  your  answer  here; 


[] 

In  how  many  source  selections  have  you  participated  throughout  your  career? 
* 

Please  write  your  answer  here; 


hnps://wrvev-npi.e(fu/«dfnin/priniAblesuivey/Mrindcx/survevid/714497 
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NPS  EmerpdM  Survey  •  BM  Protect 


S/12/14.  10:34  PM 


[]What  is  the  highest  level  of  Acquisition  Professional  Development  Program 
(APDP)  certification  that  you  hold?  * 

Please  choose  only  one  of  the  folowing: 

O  APDP/DAWIA  Level  1 
O  APDP/DAWIA  Level  2 
O  APDP/DAWIA  Level  3 
O  No  APDP  or  DAWIA  Certilication 


[] 

What  is  the  highest  level  of  education  that  you  have  attained?  * 
Please  choose  only  one  of  the  folowing: 

O  High  School  Diploma  /  GED 
O  Associates 
O  Bachelors 
O  Masters 

C?  Doctoral  /  Professional 


[]How  many  years  of  experience  do  you  have  evaluating  contractor  performance?  * 
Please  write  your  answer  here: 


[] 

What  is  your  gender? 

Please  choose  only  one  of  the  folowing: 

C)  Female 
C>  Male 


hnps://iurvey.rM>s.e<tu/ad(nin/pr(maMesurvey/u/1n(lex/»urveyid/714497 
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NPS  Enterprise  Survey  -  Bd  Protest 


5/12/14. 1(h34  PM 


[] 

We  appreciate  any  comments  or  feedback  you  can  provide  on  the  topic  of  contractor 
protests  in  government  contracting  and/or  this  survey,  (optional) 

Please  write  your  arrswer  here; 


https://survev.nps.etfu/3dmin  >'prjntdMesurvey/M,‘in(5ex/vurveyid/ 714497 
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APPENDIX  B.  SURVEY  INVITATIONS 


NAVAL 

POSTGRADUATE 

SCHOOL 

Dear  Colleague, 


You  have  been  randomly  selected  to  participate  in  a  study  of  bid  Protests.  This  research  is 
approved  by  Mr.  Elliot  Branch,  Deputy  Assistant  Secretary  of  the  Navy  for  Acquisition 
and  Procurement,  and  will  help  us  fulfill  graduation  requirements  for  our  MBA  degree. 

The  purpose  of  this  study  is  to  explore  the  effects  of  potential  bid  protests  on  acquisition 
strategies.  I  respectfully  request  your  assistance  to  complete  the  web-based  survey 
located  at  the  hyperlink  below.  Your  participation  is  completely  anonymous. 

For  your  time,  you  will  be  eligible  to  enter  a  random  drawing  for  a  new  iPad  Mini, 
16  GB.  To  enter,  follow  the  instructions  at  the  end  of  the  survey. 

Survey  Link:  https://survev.nps.edu/714497/lanq-en 

Your  participation  is  voluntary;  however,  responses  are  vital  to  conducting  valid 
research  that  represents  your  knowledge  and  experience.  Please  complete  the  survey  no 
later  than  15  FEB  2014.  The  survey  should  take  no  longer  than  30  minutes  to  complete. 

We  very  much  appreciate  any  assistance  you  can  provide.  We  would  be  more  than 
happy  to  share  our  findings  with  you  once  we  complete  our  research.  If  you  have  any 
questions  or  comments,  please  feel  free  to  contact  myself  at  jdcaland @ nps .edu  ,  or  my 
Principle  Advisor,  Lt.  Col.  Timothy  Hawkins,  PH.D.,  USAF  at 
timothv.hawkins@wku.edu  .  Questions  about  your  rights  as  a  research  subject  may  be 
addressed  to  the  Navy  Postgraduate  School  IRB  Chair,  Dr.  Larry  Shattuck,  831-656- 
2473,  Igshattu  @  nps  .edu. 


V/R 

LCDR  Suquon  Combs,  LT  Jason  Calandruccio,  LT  Brian  Colbert 
MBA  Students,  Naval  Postgraduate  School 


Principle  Advisor:  Lt.  Col.  Timothy  Hawkins,  PH.D.,  USAF 
Advisor:  E.  Cory  Yoder,  Senior  Lecturer,  Naval  Postgraduate  School. 

This  research  is  being  conducted  through  the  Naval  Postgraduate  School’s  Graduate 
School  of  Business  and  Public  Policy.  The  questionnaire  is  anonymous;  your  responses 
cannot  be  linked  to  you.  There  are  not  necessarily  “right  answers.” 


Procedures.  Your  extent  of  participation  in  this  research  involves  only  the  completion  of 
this  questionnaire. 
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Synopsis.  This  is  both  an  anonymous  and  voluntary  questionnaire.  (Please  note,  in  order 
to  obtain  consistent  and  usable  results,  it  is  important  that  you  answer  all  questions).  It 
will  take  most  respondents  approximately  30  minutes  to  complete  the  questionnaire. 

Risks  and  Benefits.  Your  participation  in  this  research  poses  no  known  risk.  You  will  be 
asked  questions  pertaining  to  the  latest  source  selection  in  which  you  participated.  There 
will  be  no  personal  benefits  beyond  having  contributed  your  expertise  to  this  important 
research.  Results  of  the  survey  will  be  used  responsibly  and  protected  against  release  to 
unauthorized  persons;  however,  there  is  minor  risk  that  data  collected  could  be 
mismanaged.  If  desired,  you  may  contact  the  researcher  above  if  you  would  like  to 
receive  a  report  of  the  results  of  the  study. 

Confidentiality  and  Privacy  Act.  All  records  of  this  study  will  be  kept  confidential  and, 
since  responses  are  anonymous,  your  privacy  will  not  be  at  risk.  No  information  will  be 
publicly  accessible  which  could  identify  you  as  a  participant.  Responses  will  be 
maintained  by  NPS  for  ten  years,  after  which  they  will  be  destroyed. 

By  taking  this  survey,  I  am  acknowledging  that  I  have  read  and  understand  this 
information,  that  I  understand  the  nature  and  purpose  of  this  study,  including  its  risks  and 
benefits,  and  that  I  agree  to  voluntarily  participate  in  this  online  survey.  I  also  understand 
that  I  may  discontinue  at  any  time  simply  by  exiting  this  website. 

Survey  Link:  https://survev.nps.edu/714497/lanq-en 
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NAVAL 

POSTGRADUATE 

SCHOOL 


Dear  Sir  or  Ma’am, 

We  recently  invited  you  to  participate  in  a  research  study  of  bid  protest  conducted 
by  the  Naval  Postgraduate  School.  We  regret  to  inform  you  that  a  raffle  for  an  iPad  Mini 
will  not  occur  due  to  ethics  regulation  constraints  offered  by  supplemental  legal  advice. 
The  research  will  proceed,  and  if  you  have  not  had  a  chance  to  participate,  please 
consider  offering  your  expertise. 

Survey  Link:  https://survey.nps.edu/714497/lang-en 

Thank  you  and  we  apologize  for  any  inconvenience. 

V/R 

LCDR  Suquon  Combs,  LT  Jason  Calandruccio,  LT  Brian  Colbert 
MBA  Students,  Naval  Postgraduate  School 

Principle  Advisor:  Lt.  Col.  Timothy  Hawkins,  PH.D.,  USAF 
Advisor:  E.  Cory  Yoder,  Senior  Lecturer,  Naval  Postgraduate  School. 
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NAVAL 

POSTGRADUATE 

SCHOOL 


Dear  Colleague, 

Thank  you  for  your  participating  in  our  research  study.  We  need  several  more 
responses  to  achieve  the  required  response  rate.  For  those  of  you  who  have  not  been  able 
to  respond  just  yet,  please  take  a  few  minutes  and  complete  the  survey.  For  those  of  you, 
who  have  already  responded,  thank  you  much! 

If  you  have  not  served  as  a  Contracting  officer  or  contract  specialist  on  a  FAR 
Part  15  source  selection,  please  reply  and  let  me  know.  I  will  remove  you  from  the  list. 
This  will  improve  the  accuracy  of  our  response  rate. 

As  a  reminder,  we  are  conducting  a  study  examining  the  effects  of  potential  bid 
protests.  In  appreciation  of  your  participation,  I  will  be  happy  to  send  you  a  brief  of  the 
study  results.  Just  reply  to  this  message  to  request  the  summary  report.  Again,  this  report 
will  contain  descriptive  statistics  based  on  collective  responses  of  all  participating 
organizations;  no  individual  response  data  will  be  published.  Your  response  will  be 
completely  anonymous.  None  of  the  information  put  in  the  survey  can  be  traced  back  to 
any  individual  nor  to  any  organization.  Thank  you  for  assisting  us  in  this  valuable  study 
required  for  the  completion  of  my  degree  at  the  Naval  Postgraduate  School. 

To  access  the  online  survey,  please  click  the  following  link: 
https://survey.nps.edu/714497/lang-en  (or  copy  and  paste  to  your  Web  browser.) 


V/R 

LT  Jason  Calandruccio 
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NAVAL 

POSTGRADUATE 

SCHOOL 


Dear  Colleague, 

We  apologize  for  not  providing  a  link  in  our  previous  e-mail.  Your  support  is  greatly 
appreeiated. 

To  aeeess  the  online  survey,  please  eliek  the  following  link: 

https://survev.nps.edu/714497/lang-en  (or  copy  and  paste  to  your  Web  browser.) 

We  still  require  several  more  responses  for  our  survey  to  achieve  a  desired  response  rate 
for  our  Deputy  Assistant  Secretary  of  the  Navy  (Acquisition  and  Procurement)  sponsored 
Naval  Postgraduate  School  MBA  Graduate  project.  If  you  have  not  had  an  opportunity 
yet,  it  is  not  too  late.  The  survey  has  been  extended  to  March  11.  It  takes  about  30 
minutes  and  will  not  only  aid  in  our  graduate  requirements,  but  will  further  our  collective 
understanding  of  our  contracting  workforce.  If  you  would  like  a  copy  of  our  research,  we 
would  be  more  than  happy  to  provide  this  to  you  once  completed.  Just  send  an  e-mail  to 
idcaland@nps.edu  or  reply  to  this  e-mail  and  we  will  ensure  you  are  on  the  list  to  receive 
it.  For  those  of  you,  who  have  already  responded,  thank  you  much! 

If  you  will  not  be  able  to  respond,  please  let  us  know  why.  This  may  make  our  response 
rate  more  accurate.  Also,  please  inform  us  if  you  are  not  a  Contracting  officer  or 
specialist  with  experience  in  at  least  one  FAR  Part  15  source  selection;  we  need  to 
remove  you  from  the  distribution  list. 

Very  Respectfully, 

LCDR  Suquon  Combs,  SC,  USN 
LT  Jason  Calandruccio,  SC,  USN 
LT  Brian  Colbert,  SC,  USN 
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APPENDIX  C.  ENDORSEMENT  BY  DASN  (AP) 


DEPARTMENT  OF  THE  NAVY 

OFFICE  OF  THE  ASSISTANT  SECRETARY 
(RESEARCH,  DEVELOPMENT  AND  ACQUISITION) 
1000  NAVY  PENTAGON 
WASHINGTON  DC  20350-1000 


JUL  1  9  2013 


FIRST  ENDORSEMENT  on  Naval  Postgraduate  School  hr  of  06  Jun  2013 

From:  Deputy  Assistant  Secretary  of  the  Navy  Acquisition  and  Procurement 
To:  Navy  Personnel  Research,  Studies,  and  Technology  (NPRST)  (PERS-14) 

Subj:  REQUEST  FOR  APPROVAL  OF  NAVY  PERSONNEL  SURVEY 

1.  Forwarded,  recommending  approval. 


Elliott  B  Branch 

Deputy  Assistant  Secretary  of  the  Navy 


Acquisition  and  Procurement 
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APPENDIX  D.  REGRESSION  OUTPUT  EROM  SPSS  VER  22 


Regression  ANTECEDENTS  OF  FEAR 


Variables  Entered/Removed^ 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

Number  of 

Source 

Selections, 

Protest  Risk, 

Dollar  Value, 

Requirement 

Criticality, 

PALT 

Planned, 

Years  of 

Experience‘s 

Enter 

a.  Dependent  Variable:  FEAR 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.361® 

.131 

.115 

7.319 

a.  Predictors:  (Constant),  Number  of  Source  Selections, 
Protest  Risk,  Dollar  Value,  Requirement  Criticality,  PALT 
Planned,  Years  of  Experience 
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ANOVA® 


Model 

df 

Mean  Square 

F 

m 

1  Regression 

2759.668 

6 

459.945 

8.587 

.000^" 

Residual 

18373.021 

343 

53.566 

Total 

21132.689 

349 

a.  Dependent  Variable:  FEAR 

b.  Predictors:  (Constant),  Number  of  Source  Selections,  Protest  Risk,  Dollar  Value, 
Requirement  Criticality,  PALT  Planned,  Years  of  Experience 


Coefficientsa 


:  1 


Unstandardized 

Standardized 

Coefficients 

Coefficients 

Constructa 

B 

Std.  Error 

Beta 

t 

Sig. 

10.995 

2.852 

3.856 

.000 

PALT  Planned 

-.259 

.091 

-.150 

-2.862 

.004 

Dollar  Value 

.721 

.238 

.155 

3.032 

.003 

Requirement 

Criticality 

.431 

.135 

.167 

3.183 

.002 

Protest  Risk 

.374 

.171 

.116 

2.190 

.029 

Years  of 

Experience 

.037 

.042 

.049 

.881 

.379 

Number  of 

Source  Selections 

-.267 

.104 

-.142 

-2.558 

.011 

a.  Dependent  Variable:  FEAR 
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Regression  FEAR  TO  TEE 


Variables  Entered/Removed^ 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

FEAR 

Stepwise 
(Criteria: 
Probability-of- 
F-to-enter  <= 
.050, 

Probability-of- 
F-to-remove 
>=  .100). 

a.  Dependent  Variable:  Quality  of  Evaluation  Factors 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.253® 

.064 

.061 

4.046 

a.  Predictors:  (Constant),  FEAR 


ANOVA® 


Model 

Sum  of 
Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

390.783 

1 

390.783 

23.869 

.OOO^" 

Residual 

5697.434 

348 

16.372 

Total 

6088.217 

349 

a.  Dependent  Variable:  Technical  Evaluation  Effectiveness 

b.  Predictors:  (Constant),  FEAR 
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Coefficients^ 


Unstandardized  Coefficients 

Standardized 

Coefficients 

Model 

B 

Std.  Error 

Beta 

t 

Sig. 

1  (Constant) 

16.343 

.590 

27.719 

.000 

FEAR 

-.136 

.028 

-.253 

-4.886 

.000 

a.  Dependent  Variable:  Quality  of  Evaluation  Factors 


Logistic  Regression  FEAR  to  PALT  Actuai 


Case  Processing  Summary 


Unweighted  Cases® 

N 

Percent 

Selected  Cases  Included  in 

Analysis 

350 

100.0 

Missing  Cases 

0 

.0 

Total 

350 

100.0 

Unselected  Cases 

0 

.0 

Total 

350 

100.0 

a.  If  weight  is  in  effect,  see  classification  table  for  the  total 
number  of  cases. 


Dependent  Variable 
Encoding 


Original  Value 

Internal  Value 

.000 

0 

1.000 

1 

Block  0:  Beginning  Block 
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Classifical 

tion  Table®  *’ 

Predicted 

PALT_ACTUAL 

Percentage 

Observed 

.000 

1.000 

Correct 

Step  0 

PALT_ACTUA  .000 

0 

145 

.0 

L  1 .000 

0 

100.0 

Overall  Percentage 

58.6 

a.  Constant  is  included  in  the  model. 

b.  The  cut  value  is  .500 


Variables  in  the  Equal 

tion 

B 

S.E. 

Wald 

df 

Sig. 

Exp(B) 

Step  0  Constant 

.346 

.109 

10.184 

1 

.001 

1.414 

Variables  not  in  the  Equation 


Score 

df 

Sig. 

Step  0  Variables  FEAR 

12.318 

1 

.000 

Overall  Statistics 

12.318 

1 

.000 

Block  1 :  Method  =  Enter 


Omnibus  ‘ 

rests  of  Model  Coefficients 

Chi-square 

df 

Sig. 

Step  1  Step 

1 2.454 

1 

.000 

Block 

1 2.454 

1 

.000 

Model 

1 2.454 

1 

.000 
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Model  Summary 


Step 

-2  Log 
likelihood 

Cox  &  Snell  R 
Square 

Nagelkerke  R 
Square 

1 

462.412® 

.035 

.047 

a.  Estimation  terminated  at  iteration  number  3 
because  parameter  estimates  changed  by  less  than 
.001. 


H 

losmer  and  Lemeshow  Test 

Step 

Chi-square 

df 

Sig. 

1 

1 1 .607 

8 

.170 

Classification  Tabie^ 


Observed 

Predicted 

PALT_ACTUAL 

Percentage 

Correct 

.000 

1.000 

Step  1 

PALT_ACTUA  .000 

39 

106 

26.9 

L  1 .000 

35 

170 

82.9 

Overall  Percentage 

59.7 

a.  The  cut  value  is  .500 


Contingency  Table  for  Hosmer  and  Lemeshow  Test 


PALT_ACTUAL  =  .000 

PALT_ACTUAL  =  1 .000 

Observed 

Observed 

Total 

Step  1  1 

22 

19.683 

12 

14.317 

2 

17 

20.943 

23 

19.057 

3 

20 

18.313 

18 

19.687 

38 

4 

17 

15.991 

19 

20.009 

36 

5 

10 

16.286 

30 

23.714 

40 

6 

17 

14.323 

21 

23.677 

38 

7 

14 

13.845 

25 

25.155 

39 

8 

14 

10.341 

17 

20.659 

31 

9 

12 

10.715 

24 

25.285 

36 

10 

2 

4.561 

16 

13.439 

18 
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Variables  in  the  Equation 


B 

S.E. 

Wald 

df 

1 

1 

Lower 

Step  FEAR 
“1  ^  Constant 

.050 

-.630 

.014 

.300 

1 1 .996 

4.419 

1 

1 

1.051 

.533 

1.022 

1.082 

a.  Variable(s)  entered  on  step  1 :  FEAR. 


Regression  FEAR  to  SSMA 


Variables  Entered/Removed^ 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

FEAR‘D 

Enter 

a.  Dependent  Variable:  SSMA 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.059® 

.003 

.001 

6.65306 

a.  Predictors:  (Constant),  FEAR 


ANOVA® 


Model 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Sig. 

53.021 

1 

53.021 

1.198 

.275“" 

Residual 

15403.596 

348 

44.263 

Total 

15456.617 

349 

a.  Dependent  Variable:  SSMA 
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b.  Predictors:  (Constant),  FEAR 


Coefficients^ 


Unstandardized 

Coefficients 

Standardized 

Coefficients 

Model 

B 

Std.  Error 

Beta 

t 

Sig. 

1  (Constant) 

34.050 

.969 

35.123 

.000 

FEAR 

-.050 

.046 

-.059 

-1.094 

.275 

a.  Dependent  Variable:  SSMA 

Logistic  Regression  FEAR  HI/LO  to  LPTA  Appropriateness 


Case  Processing  Summary 


Unweighted  Cases® 

N 

Percent 

Selected  Cases  Included  in  Analysis 

133 

38.0 

Missing  Cases 

217 

62.0 

Total 

350 

100.0 

Unselected  Cases 

0 

.0 

Total 

350 

100.0 

a.  If  weight  is  in  effect,  see  classification  table  for  the  total 
number  of  cases. 


Dependent  Variable  Encoding 


Original  Value 

Internal  Value 

Low 

0 

High 

1 

Block  0:  Beginning  Block 
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Classification  Table 


a,b 


Observed 

Predictec 

LPTA  Appropriateness 

Percentage 

Correct 

Low 

High 

Step  0  LPTA  Low 

120 

0 

100.0 

Inappropriateness  High 

13 

0 

.0 

Overall  Percentage 

90.2 

a.  Constant  is  included  in  the  model. 

b.  The  cut  value  is  .500 


Variables  in  the  Equal 

tion 

B 

S.E. 

Wald 

df 

Sig. 

Exp(B) 

Step  0  Constant 

-2.223 

.292 

57.939 

1 

.000 

.108 

Variables  not  in  the  Equation 


Score 

df 

Step  0  Variables  FEAR_LPTA_USE 

D 

6.861 

1 

.009 

Overall  Statistics 

6.861 

1 

.009 

Block  1 :  Method  =  Enter 


Omnibus  Tests  of  Model  Coefficients 


df 

6.559 

1 

.010 

Block 

6.559 

1 

.010 

Model 

6.559 

1 

.010 

Mod 

lei  Summary 

H 

-2  Log 

Cox  &  Snell  R 

Nagelkerke  R 

SB 

likelihood 

Square 

Square 

1 

78.587® 

.048 

.102 

a.  Estimation  terminated  at  iteration  number  6  because 
parameter  estimates  changed  by  less  than  .001 . 
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Classification  Table^ 


Predicted 

LPTA  Appropriateness 

Percentag 

Observed 

Low 

High 

e  Correct 

Step  1  LPTA 

Low 

120 

0 

100.0 

Inappropriateness 

High 

13 

0 

.0 

Overall  Percentage 

90.2 

a.  The  cut  value  is  .500 


Variables  in  the  Eq 

luation 

B 

S.E. 

Wald 

df 

Sig. 

Exp(B) 

Step  FEAR_LPTA_USED 

1.542 

.632 

5.956 

1 

.015 

4.673 

1  ^  Constant 

-3.008 

.512 

34.493 

1 

.000 

.049 

Logistic  Regression  FEAR  HI/LO  to  T/0  Appropriateness 


Case  Processing  Summary 


Unweighted  Cases® 

N 

Percent 

Selected  Cases  Included  in  Analysis 

174 

49.7 

Missing  Cases 

176 

50.3 

Total 

350 

100.0 

Unselected  Cases 

0 

.0 

Total 

350 

100.0 

a.  If  weight  is  in  effect,  see  classification  table  for  the  total 
number  of  cases. 


Dependent  Variable  Encoding 


Original  Value 

Internal  Value 

Appropriate 

0 

Not  Appropriate 

1 
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Block  0:  Beginning  Block 


Classification  Table^’^’ 


Observed 

Predicted 

TRADE-OFF_BINARY 

Percentage 

Correct 

Appropriate 

Not 

Appropriate 

Step  0  TRADE-  Appropriate 

164 

0 

100.0 

OFF_BINARY  Not 

Appropriate 

10 

0 

.0 

Overall  Percentage 

94.3 

a.  Constant  is  included  in  the  model. 

b.  The  cut  value  is  .500 


Variables  in  the  Equal 

tion 

B 

S.E. 

Wald 

df 

-2.797 

.326 

73.751 

1 

.000 

.061 

Variables  not  in  the  Equation 


Score 

df 

Sig. 

Step  0  Variables  FEAR_BINARY_T 

0 

1.835 

1 

.176 

Overall  Statistics 

1.835 

1 

.176 

Block  1 :  Method  =  Enter 


Omnibus  Tests  of  Model  Coefficients 


df 

Step  1  Step 

1.786 

1 

.181 

Block 

1.786 

1 

.181 

Model 

1.786 

1 

.181 
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Mod 

lei  Summary 

H 

-2  Log 

Cox  &  Snell  R 

Nagelkerke  R 

SB 

likelihood 

Square 

Square 

1 

74.758® 

.010 

.029 

a.  Estimation  terminated  at  iteration  number  6  because 
parameter  estimates  changed  by  less  than  .001 . 


Hosmer  and  Lemeshow  Test 


Step 

Chi-square 

df 

Sig. 

1 

.000 

0 

Contingency  Table  for  Hosmer  and  Lemeshow  Test 


TRADE-OFF_BINARY  = 
Appropriate 

TRADE-OFF_BINARY  = 
Not  Appropriate 

Total 

Observed 

Observed 

101 

101.000 

4 

4.000 

105 

2 

63 

63.000 

6 

6.000 

69 

Classification  Table^ 


Observed 

Predicted 

TRADE-OFF_BINARY 

Percentage 

Correct 

Appropriate 

Not 

Appropriate 

Step  1  TRADE-  Appropriate 

164 

0 

100.0 

OFF_BINARY  Not 

Appropriate 

10 

0 

.0 

Overall  Percentage 

94.3 

a.  The  cut  value  is  .500 
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Variab 


es  in  the  Equation 


B 

S.E. 

Wald 

df 

1 

H 

Lower 

Step  FEAR_BINARY_TO 

-.877 

.665 

1.740 

1 

.187 

.416 

.113 

1.531 

1  ^  Constant 

2.351 

.427 

30.289 

1 

.000 

.095 

a.  Variable(s)  entered  on  step  1 :  FEAR_BINARY_TO. 

Regression  FEAR  HI/LO  to  Satisfaction  with  Discussions 


Variabies  Entered/Removed^ 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

Enter 

a.  Dependent  Variable:  DISAT_DISC 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.111® 

.012 

.010 

1.710 

a.  Predictors:  (Constant),  HIGH_FEAR 


ANOVA® 


Model 

df 

Mean 

Square 

F 

m 

12.781 

1 

12.781 

4.370 

.037'^ 

Residual 

1017.779 

348 

2.925 

Total 

1030.560 

349 

a.  Dependent  Variable:  DISAT_DISC 

b.  Predictors:  (Constant),  FIIGFI_FEAR 
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Coefficients^ 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

3.087 

.130 

23.740 

.000 

HIGH_FEAR 

.382 

.183 

.111 

2.091 

.037 

a.  Dependent  Variable:  DISAT_DISC 

Regression  FEAR  to  Hypotheticai  Change  in  Discussions 


Variabies  Entered/Removed^ 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

Enter 

a.  Dependent  Variable:  if  there  was  no  ability  to  protest 
question 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.316® 

.100 

.097 

1 .93920 

a.  Predictors:  (Constant),  HIGH_FEAR 


ANOVA^ 


Model 

Sum  of 
Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

145.309 

1 

145.309 

38.641 

.OOO*" 

Residual 

1308.659 

348 

3.761 

Total 

1453.969 

349 

a.  Dependent  Variable:  if  there  was  no  ability  to  protest  question 

b.  Predictors:  (Constant),  HIGH_FEAR 
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Coefficients^ 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

2.723 

.147 

18.466 

.000 

HIGH_FEAR 

1.289 

.207 

.316 

6.216 

.000 

a.  Dependent  Variable:  if  there  was  no  ability  to  protest  question 


Logistic  Regression  FEAR  to  Discussions  Heid  Y/N 


Case  Processing  Summary 


Unweighted  Cases® 

N 

Percent 

Selected  Cases  Included  in  Analysis 

350 

100.0 

Missing  Cases 

0 

.0 

Total 

350 

100.0 

Unselected  Cases 

0 

.0 

Total 

350 

100.0 

a.  If  weight  is  in  effect,  see  classification  table  for  the  total 
number  of  cases. 


Dependent  Variable  Encoding 


Original  Value 

Internal  Value 

No 

0 

Yes 

1 

183 


Block  0:  Beginning  Block 


Classification  Table^*’ 


Observed 

Predicted  | 

Discussions 

Percentage 

Correct 

No 

Yes 

Step  0  Discussions  No 

0 

147 

.0 

Yes 

0 

203 

100.0 

Overall  Percentage 

58.0 

a.  Constant  is  included  in  the  model. 

b.  The  cut  value  is  .500 


Variables  in  the  Equal 

tion 

B 

S.E. 

Wald 

df 

Sig. 

Exp(B) 

Step  0  Constant 

.323 

.108 

8.883 

1 

.003 

1.381 

Variables  not  in  the  Equation 


Score 

df 

Sig. 

Step  0  Variables  FEAR 

4.356 

1 

.037 

Overall  Statistics 

4.356 

1 

.037 

Block  1 :  Method  =  Enter 


Omnibus  Tests  of  Model  Coefficients 


Chi-square 

df 

Sig. 

Step  1  Step 

4.370 

1 

.037 

Block 

4.370 

1 

.037 

Model 

4.370 

1 

.037 
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Mod 

lei  Summary 

H 

-2  Log 

Cox  &  Snell  R 

Nagelkerke  R 

SB 

likelihood 

Square 

Square 

1 

471 .834® 

.012 

.017 

a.  Estimation  terminated  at  iteration  number  3  because 
parameter  estimates  changed  by  less  than  .001 . 


Classification  Table^ 


Observed 

Predicted  | 

Discussions 

Percentage 

Correct 

No 

Yes 

Step  1  Discussions  No 

17 

130 

11.6 

Yes 

17 

186 

91.6 

Overall  Percentage 

58.0 

a.  The  cut  value  is  .500 


Variables  in  the  Equal 

[ion 

B 

S.E. 

Wald 

df 

l^lsH 

.029 

.014 

4.315 

1 

.038 

1.030 

Constant 

-.251 

.295 

.722 

1 

.396 

.778 

a.  Variable(s)  entered  on  step  1 :  FEAR. 

Regression  FEAR  to  Oral  Pres.  Appropriateness 


Variables  Entered/Removed^ 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

FEAR‘D 

Enter 

a.  Dependent  Variable:  ORAL_PRESENTATIONS 

b.  All  requested  variables  entered. 
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Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.060® 

.004 

.001 

2.09783 

a.  Predictors:  (Constant),  FEAR 


ANOVA® 


Model 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Sig. 

5.587 

1 

5.587 

1.269 

.261 

Residual 

1531.510 

348 

4.401 

Total 

1537.097 

349 

a.  Dependent  Variable;  ORAL_PRESENTATIONS 

b.  Predictors:  (Constant),  FEAR 


Coefficients^ 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

5.023 

.306 

16.433 

.000 

FEAR 

-.016 

.014 

-.060 

-1.127 

.261 

a.  Dependent  Variable:  ORAL_PRESENTATIONS 

Regression  FEAR  to  Resources 


Variabies  Entered/Removed^ 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

FEAR*" 

Enter 

a.  Dependent  Variable; 

NUMBER_OF_PEOPLE_ON_SS 

b.  All  requested  variables  entered. 
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Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.277® 

.077 

.074 

.28028 

a.  Predictors:  (Constant),  FEAR 


ANOVA® 


Model 

df 

Mean 

Square 

F 

m 

2.280 

1 

2.280 

29.022 

.000'^ 

Residual 

27.338 

348 

.079 

Total 

29.618 

349 

a.  Dependent  Variable:  NUMBER_OF_PEOPLE_ON_SS 

b.  Predictors:  (Constant),  FEAR 


Coefficients^ 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

m 

B 

Std.  Error 

Beta 

1  (Constant) 

.632 

.041 

15.476 

.000 

FEAR 

.010 

.002 

.277 

5.387 

.000 

Regression  FEAR  Hi/Lo  to  Resources 


Variabies  Entered/Removed^ 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

Enter 
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Variable; 


a.  Dependent 
NUMBER_OF_PEOPLE_ON_SS 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.257® 

.066 

.063 

.28197 

a.  Predictors:  (Constant),  HIGH_FEAR 


ANOVA® 


Model 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Sig. 

1.949 

1 

1.949 

24.513 

.000“" 

Residual 

27.669 

348 

.080 

Total 

29.618 

349 

a.  Dependent  Variable:  NUMBER_OF_PEOPLE_ON_SS 

b.  Predictors:  (Constant),  FIIGFI_FEAR 


Coefficients^ 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

.761 

.021 

35.509 

.000 

HIGH_FEAR 

.149 

.030 

.257 

4.951 

.000 

a.  Dependent  Variable:  NUMBER_OF_PEOPLE_ON_SS 


Logistic  Regression  FEAR  to  More  Resources  than  needed 
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Case  Processing  Summary 


Unweighted  Cases® 

N 

Percent 

Selected  Cases  Included  in  Analysis 

350 

100.0 

Missing  Cases 

0 

.0 

Total 

350 

100.0 

Unselected  Cases 

0 

.0 

Total 

350 

100.0 

a.  If  weight  is  in  effect,  see  classification  table  for  the  total 
number  of  cases. 


Dependent  Variable  Encoding 


Internal  Value 

0 

0 

1 

1 

Block  0:  Beginning  Block 


Classification  Table^*’ 


Observed 

Predicted  | 

RESOURCES 

Percentage 

Correct 

0 

1 

StepO  RESOURCES  0 

240 

0 

100.0 

1 

110 

0 

.0 

Overall  Percentage 

68.6 

a.  Constant  is  included  in  the  model. 

b.  The  cut  value  is  .500 
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Variables  in  the  Equal 

tion 

B 

S.E. 

Wald 

df 

Sig. 

Exp(B) 

Step  0  Constant 

-.780 

.115 

45.909 

1 

.000 

.458 

Variables  not  in  the  Equation 


Score 

df 

Step  0  Variables  FEAR 

.774 

1 

.379 

Overall  Statistics 

.774 

1 

.379 

Block  1 :  Method  =  Enter 


Omnibus  Tests  of  Model  Coefficients 


df 

Step  1  Step 

.776 

1 

.378 

Block 

.776 

1 

.378 

Model 

.776 

1 

.378 

Mod 

lei  Summary 

-2  Log 

Cox  &  Snell  R 

Nagelkerke  R 

Step 

likelihood 

Square 

Square 

1 

434.965® 

.002 

.003 

a.  Estimation  terminated  at  iteration  number  3  because 
parameter  estimates  changed  by  less  than  .001 . 


Hosmer  and  Lemeshow  Test 


Step 

Chi-square 

df 

Sig. 

1 

1 1 .643 

8 

.168 
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Contingency  Table  for  Hosmer  and  Lemeshow  Test 


RESOUF 

ICES  =  0 

RESOUF 

ICES  =  1 

Total 

Observed 

Observed 

Step  1  1 

28 

24.577 

6 

9.423 

34 

2 

27 

28.440 

13 

1 1 .560 

40 

3 

29 

26.675 

9 

1 1 .325 

38 

4 

23 

24.970 

13 

1 1 .030 

36 

5 

25 

27.406 

15 

12.594 

40 

6 

21 

25.763 

17 

12.237 

38 

7 

23 

26.230 

16 

12.770 

39 

8 

25 

20.680 

6 

10.320 

31 

9 

26 

23.663 

10 

12.337 

36 

10 

13 

1 1 .595 

5 

6.405 

18 

Classification  Table^ 


Observed 

Predicted  | 

RESOURCES 

Percentage 

Correct 

0 

1 

Step  1  RESOURCES  0 

240 

0 

100.0 

1 

110 

0 

.0 

Overall  Percentage 

68.6 

a.  The  cut  value  is  .500 


Variables  in  the  Equation 


B 

S.E. 

Wald 

df 

■ 

1 

Lower 

Step  FEAR 

.013 

.015 

.773 

1 

.379 

1.013 

.984 

1.043 

1  ^  Constant 

-1.040 

.319 

10.608 

1 

.001 

.353 

a.  Variable(s)  entered  on  step  1 :  FEAR. 
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Regression  FEAT  to  TCI 


Variables  Entered/Removed^ 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

FEAR 

Stepwise 
(Criteria: 
Probability-of- 
F-to-enter  <= 
.050, 

Probability-of- 
F-to-remove  >= 
.100). 

a.  Dependent  Variable:  TCI 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.318® 

.101 

.099 

.32938 

a.  Predictors:  (Constant),  FEAR 


ANOVA® 


Model 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Sig. 

4.246 

1 

4.246 

39.135 

.000“" 

Residual 

37.755 

348 

.108 

Total 

42.001 

349 

a.  Dependent  Variable:  TCI 

b.  Predictors:  (Constant),  FEAR 
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Coefficients^ 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

1.080 

.048 

22.505 

.000 

FEAR 

.014 

.002 

.318 

6.256 

.000 

a.  Dependent  Variable:  TCI 


Regression  FEAR  to  Source  Seiection  Pian 


Variabies  Entered/Removed^ 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

FEAR*" 

Enter 

a.  Dependent  Variable:  TCI4 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.290® 

.084 

.082 

.58979 

a.  Predictors:  (Constant),  FEAR 


ANOVA® 


Model 

df 

Mean 

Square 

F 

m 

11.147 

1 

11.147 

32.045 

.000*" 

Residual 

121.051 

348 

.348 

Total 

132.198 

349 

a.  Dependent  Variable:  TCI4 

b.  Predictors:  (Constant),  FEAR 
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Coefficients^ 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

m 

B 

Std.  Error 

Beta 

1  (Constant) 

1.110 

.086 

12.918 

.000 

FEAR 

.023 

.004 

.290 

5.661 

.000 

a.  Dependent  Variable:  TCI4 

Regression  FEAR  to  Debriefing  Script 


Variabies  Entered/Removed^ 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

FEAR‘D 

Enter 

a.  Dependent  Variable:  TCI5 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.280® 

.078 

.076 

.75232 

a.  Predictors:  (Constant),  FEAR 


ANOVA® 


Model 

df 

Mean 

Square 

F 

m 

16.707 

1 

16.707 

29.519 

.ooo*" 

Residual 

196.964 

348 

.566 

Total 

213.672 

349 

a.  Dependent  Variable:  TCI5 

b.  Predictors:  (Constant),  FEAR 
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Coefficients^ 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

m 

B 

Std.  Error 

Beta 

1  (Constant) 

.729 

.110 

6.651 

.000 

FEAR 

.028 

.005 

.280 

5.433 

.000 

a.  Dependent  Variable:  TCI5 


Regression  FEAR  to  Coior/Rating  Chart 


Variabies  Entered/Removed^ 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

FEAR‘D 

Enter 

a.  Dependent  Variable:  TCI9 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.195® 

.038 

.035 

.77260 

a.  Predictors:  (Constant),  FEAR 


ANOVA® 


Model 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Sig. 

8.221 

1 

8.221 

13.772 

.000“" 

Residual 

207.727 

348 

.597 

Total 

215.948 

349 

a.  Dependent  Variable:  TCI9 

b.  Predictors:  (Constant),  FEAR 
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Coefficients^ 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

.490 

.113 

4.350 

.000 

FEAR 

.020 

.005 

.195 

3.711 

.000 

a.  Dependent  Variable:  TCI9 

Regression 


Variabies  Entered/Removed^ 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

FEAR‘D 

Enter 

a.  Dependent  Variable:  TCI10 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.196® 

.039 

.036 

.69098 

a.  Predictors:  (Constant),  FEAR 


ANOVA® 


Model 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Sig. 

6.665 

1 

6.665 

13.959 

.ooo'" 

Residual 

166.154 

348 

.477 

Total 

172.819 

349 
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a.  Dependent  Variable:  TCI10 

b.  Predictors:  (Constant),  FEAR 


Coefficients^ 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

m 

B 

Std.  Error 

Beta 

1  (Constant) 

.760 

.101 

7.548 

.000 

FEAR 

.018 

.005 

.196 

3.736 

.000 

a.  Dependent  Variable:  TCI10 


Regression  FEAR  to  TCP 


Variabies  Entered/Removed^ 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

Fear  N=270‘' 

Enter 

a.  Dependent  Variable:  Transaction  Cost  Using  Sqrt 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.143® 

.021 

.017 

.45284 

a.  Predictors:  (Constant),  Fear  N=270 
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ANOVA® 


Model 

df 

Mean 

Square 

F 

m 

1.155 

1 

1.155 

5.631 

.01 8*^ 

Residual 

54.957 

268 

.205 

Total 

56.112 

269 

a.  Dependent  Variable:  Transaction  Cost  Using  Sqrt 

b.  Predictors:  (Constant),  Fear  N=270 


Coefficients^ 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

4.993 

.078 

63.838 

.000 

Fear  N=270 

.009 

.004 

.143 

2.373 

.018 

a.  Dependent  Variable:  Transaction  Cost  Using  Sqrt 


Logistic  Regression  FEAT  to  AC 


Case  Processing  Summary 


Unweighted  Cases® 

N 

Percent 

Selected  Cases  Included  in  Analysis 

350 

100.0 

Missing  Cases 

0 

.0 

Total 

350 

100.0 

Unselected  Cases 

0 

.0 

Total 

350 

100.0 

a.  If  weight  is  in  effect,  see  classification  table  for  the  total 
number  of  cases. 
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Dependent  Variable  Encoding 


Original  Value 

Internal  Value 

0 

0 

1 

1 

Block  0:  Beginning  Block 


Classification  Table^’^’ 


Observed 

Predicted 

Binary.. .if  more  were 
awarded 

Percentage 

Correct 

0 

1 

Step  0  Binary.. .if  more  0 

331 

0 

100.0 

were  awarded  1 

19 

0 

.0 

Overall  Percentage 

94.6 

a.  Constant  is  included  in  the  model. 

b.  The  cut  value  is  .500 


Variables  in  the  Equal 

tion 

B 

S.E. 

Wald 

df 

l^lsH 

-2.858 

.236 

146.737 

1 

.000 

.057 

Variables  not  in  the  Equation 


Score 

df 

Sig. 

Step  0  Variables  FEAR 

.863 

1 

.353 

Overall  Statistics 

.863 

1 

.353 
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Block  1 :  Method  =  Enter 


Omnibus  Tests  of  Model  Coefficients 


Chi-square 

df 

Sig. 

Step  1  Step 

.872 

1 

.350 

Block 

.872 

1 

.350 

Model 

.872 

1 

.350 

Mod 

lei  Summary 

Step 

-2  Log 
likelihood 

Cox  &  Snell  R 
Square 

Nagelkerke  R 
Square 

1 

146.790® 

.002 

.007 

a.  Estimation  terminated  at  iteration  number  6  because 
parameter  estimates  changed  by  less  than  .001 . 


Hosmer  and  Lemeshow  Test 


Step 

Chi-square 

df 

Sig. 

1 

3.247 

8 

.918 

Contingency  Table  for  Hosmer  and  Lemeshow  Test 


Binary.. .if  more  were 
awarded  =  0 

Binary.. .if  more  were 
awarded  =  1 

Total 

Observed 

Expected 

Observed 

Expected 

33 

32.751 

1 

1.249 

34 

2 

38 

38.338 

2 

1.662 

40 

3 

37 

36.270 

1 

1.730 

38 

4 

34 

34.220 

2 

1.780 

36 

5 

36 

37.853 

4 

2.147 

40 

6 

36 

35.816 

2 

2.184 

38 

7 

38 

36.641 

1 

2.359 

39 

8 

29 

29.027 

2 

1.973 

31 

9 

33 

33.493 

3 

2.507 

36 

10 

17 

16.591 

1 

1.409 

18 
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Classification  Table^ 


Observed 

Predicted 

Binary.. .if  more  were 
awarded 

Percentage 

Correct 

0 

1 

Step  1  Binary.. .if  more  0 

331 

0 

100.0 

were  awarded  1 

19 

0 

.0 

Overall  Percentage 

94.6 

a.  The  cut  value  is  .500 


Variables  in  the  Equation 


B 

S.E. 

Wald 

df 

Sig. 

m 

X 

95%  C.I.for 
EXP(B) 

Lower 

Upper 

.029 

.031 

.857 

1 

.355 

1.029 

.968 

1.094 

-3.445 

.699 

24.276 

1 

.000 

.032 

a.  Variable(s)  entered  on  step  1 :  FEAR. 


Regression  FEAR  to  CO  AUTH 


Variables  Entered/Removed^ 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

FEAR‘D 

Enter 

a.  Dependent  Variable:  Contracting  officer  Authority 

b.  All  requested  variables  entered. 
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Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.090® 

.008 

.005 

6.005 

a.  Predictors:  (Constant),  FEAR 


ANOVA® 


Model 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Sig. 

102.573 

1 

102.573 

2.844 

.093“" 

Residual 

12550.696 

348 

36.065 

Total 

12653.269 

349 

a.  Dependent  Variable:  Contracting  officer  Authority 

b.  Predictors:  (Constant),  FEAR 


Coefficients^ 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

26.419 

.875 

30.190 

.000 

FEAR 

-.070 

.041 

-.090 

-1.686 

.093 

a.  Dependent  Variable:  Contracting  officer  Authority 


Regression  FEAR  Hi/Lo  to  CO  AUTH 


Variabies  Entered/Removed^ 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

Enter 

a.  Dependent  Variable:  Contracting  officer  Authority 

b.  All  requested  variables  entered. 
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Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.006® 

.000 

-.003 

6.030 

a.  Predictors:  (Constant),  HIGH_FEAR 


ANOVA® 


Model 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Sig. 

.424 

1 

.424 

.012 

.91 4“" 

Residual 

12652.844 

348 

36.359 

Total 

12653.269 

349 

a.  Dependent  Variable:  Contracting  officer  Authority 

b.  Predictors:  (Constant),  HIGH_FEAR 


Coefficients^ 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

25.081 

.458 

54.709 

.000 

HIGH_FEAR 

-.070 

.645 

-.006 

-.108 

.914 

a.  Dependent  Variable:  Contracting  officer  Authority 
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Regression  TEE/SSMA  to  CP 


Variables  Entered/Removed^ 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

Source 

Selection 

Method 

Appropriatenes 
s.  Tech  Eval 
Effectiveness*^ 

Enter 

a.  Dependent  Variable:  Contractor  Performance 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.350® 

.123 

.118 

7.671 

a.  Predictors:  (Constant),  Source  Selection  Method 
Appropriateness,  Tech  Eval  Effectiveness 


ANOVA® 


Model 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Sig. 

2854.174 

2 

1427.087 

24.255 

.000*" 

Residual 

20416.423 

347 

58.837 

Total 

23270.597 

349 

a.  Dependent  Variable:  Contractor  Performance 

b.  Predictors:  (Constant),  Source  Selection  Method  Appropriateness,  Tech  Eval 
Effectiveness 
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Coefficients^ 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

17.765 

2.245 

7.912 

.000 

Tech  Eval 

Effectiveness 

.327 

.102 

.167 

3.209 

.001 

Source 

Selection 

Method 

Appropriateness 

.328 

.064 

.268 

5.140 

.000 

a.  Dependent  Variable:  Contractor  Performance 

Regression  LPTA  Appr  to  CP 


Variabies  Entered/Removed^ 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

LPTA 

Appropriatenes 
s  (4  or  less) 
when  LPTA 

used‘s 

Enter 

a.  Dependent  Variable:  CP 

b.  All  requested  variables  entered. 
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Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.222® 

.049 

.042 

7.84804 

a.  Predictors:  (Constant),  LPTA  Appropriateness  (4  or  less)  when 
LPTA  used 


ANOVA® 


Model 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Sig. 

419.203 

1 

419.203 

6.806 

.01  o'" 

Residual 

8068.526 

131 

61.592 

Total 

8487.729 

132 

a.  Dependent  Variable:  CP 

b.  Predictors:  (Constant),  LPTA  Appropriateness  (4  or  less)  when  LPTA  used 


Coefficients^ 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

33.819 

.766 

44.156 

.000 

LPTA 

Appropriateness 
(4  or  less)  when 
LPTA  used 

-4.355 

1.669 

-.222 

-2.609 

.010 

a.  Dependent  Variable:  CP 

Regression 


Variabies  Entered/Removed^ 
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Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

LPTA_Appropri 

ateness'^ 

Enter 

a.  Dependent  Variable:  CP 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.238® 

.057 

.049 

7.81857 

a.  Predictors:  (Constant),  LPTA_Appropriateness 


ANOVA® 


Model 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Sig. 

479.698 

1 

479.698 

7.847 

.006“" 

Residual 

8008.032 

131 

61.130 

Total 

8487.729 

132 

a.  Dependent  Variable:  CP 

b.  Predictors:  (Constant),  LPTA_Appropriateness 


Coefficients^ 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std. 

Error 

Beta 

1  (Constant) 

25.794 

2.626 

9.821 

.000 

LPTA_Appropriateness 

1.237 

.442 

.238 

2.801 

.006 

a.  Dependent  Variable:  CP 

Regression  CP  to  BS 
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Variables  Entered/Removed^ 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

Contractor 

Performance‘s 

Enter 

a.  Dependent  Variable:  BS 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.665® 

.442 

.441 

2.61931 

a.  Predictors:  (Constant),  Contractor  Performance 


ANOVA® 


Model 

df 

Mean 

Square 

F 

m 

1893.312 

1 

1893.312 

275.962 

.000‘s 

Residual 

2387.546 

348 

6.861 

Total 

4280.857 

349 

a.  Dependent  Variable:  BS 

b.  Predictors:  (Constant),  Contractor  Performance 


Coefficients^ 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

m 

B 

Std.  Error 

Beta 

1  (Constant) 

6.506 

.585 

11.121 

.000 

Contractor 

Performance 

.285 

.017 

.665 

16.612 

.000 

a.  Dependent  Variable:  BS 
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